Module Mapred_io_bgreader

module Mapred_io_bgreader: sig .. end

Background reader for Mapred_io

This module reads the blocks of a file in a background kernel thread, for increasing the resource utilization of mapred tasks. While the task is doing computations, the kernel thread can already read the next blocks from the filesystem.

type bgreader

val create : Mapred_fs.filesystem ->
       string -> int64 -> int64 -> int -> bgreader

create fs name startblock endblock bufnum

The new reader uses a copy of the client fs for the fs access. The file is name, which is read from startblock to endblock-1. bufnum is the length of the buffer in blocks (for each of the two buffers, foreground and background).

After create the first returned buffer is empty. Call read to read the first blocks.

val extend : bgreader -> int64 -> unit

extend bgreader endblock: Sets a new endblock (higher value than the old endblock)

val read : bgreader -> int

read bgreader: reads the next blocks into the buffer. Returns the number of blocks - 0 if at eof

val buffer : bgreader -> Netsys_mem.memory

Returns the buffer. Note that the module uses double buffering: There are actually two buffers, one foreground buffer, and one background buffer. This function returns the current foreground buffer which is not touched anymore by the reader. The two buffers are normally swapped when read is called.

val buffer_block : bgreader -> int64

Returns the block position of buffer

val buffer_len : bgreader -> int

Returns the length of buffer in blocks

val close : bgreader -> unit

Closes the reader

This web site is published by Informatikbüro Gerd Stolpmann

Plasma	GitLab	Archive
Projects	Blog	Knowledge