Plasma GitLab Archive
Projects Blog Knowledge

Module Mapred_io_bgreader

module Mapred_io_bgreader: sig .. end
Background reader for Mapred_io

This module reads the blocks of a file in a background kernel thread, for increasing the resource utilization of mapred tasks. While the task is doing computations, the kernel thread can already read the next blocks from the filesystem.
type bgreader 
val create : Mapred_fs.filesystem ->
string -> int64 -> int64 -> int -> bgreader
create fs name startblock endblock bufnum

The new reader uses a copy of the client fs for the fs access. The file is name, which is read from startblock to endblock-1. bufnum is the length of the buffer in blocks (for each of the two buffers, foreground and background).

After create the first returned buffer is empty. Call read to read the first blocks.

val extend : bgreader -> int64 -> unit
extend bgreader endblock: Sets a new endblock (higher value than the old endblock)
val read : bgreader -> int
read bgreader: reads the next blocks into the buffer. Returns the number of blocks - 0 if at eof
val buffer : bgreader -> Netsys_mem.memory
Returns the buffer. Note that the module uses double buffering: There are actually two buffers, one foreground buffer, and one background buffer. This function returns the current foreground buffer which is not touched anymore by the reader. The two buffers are normally swapped when read is called.
val buffer_block : bgreader -> int64
Returns the block position of buffer
val buffer_len : bgreader -> int
Returns the length of buffer in blocks
val close : bgreader -> unit
Closes the reader
This web site is published by Informatikbüro Gerd Stolpmann
Powered by Caml