Not everything works in the same way for all operating systems. This documents describes the different levels of support for the various platforms.
Ocamlnet has been primarily developed for POSIX systems. Most libraries should work on all systems, but there are also some more special features that are designed for certain systems only.
select()
On POSIX systems, the select() system call is not used anymore. As
a general-purpose replacement Ocamlnet favors poll(). In future
versions of Ocamlnet, there will also be support for the improved
versions of poll() some systems provide, such as epoll on Linux,
kqueue on BSD, and /dev/poll on Solaris. Ocamlnet is already
prepared for this change.
The reason for banning select() are the limited capabilities of this
API. Especially, it cannot handle file descriptors whose numeric
values exceed a system-dependent limit. Although this limit is quite
high (usually 1024) there are applications that need more descriptors.
(Note that it is not sufficient to simply increase the maximum number
of descriptors a process can have - the select() call is still
restricted in the mentioned sense, and this seems to be unfixable.)
The poll() interface does not have this problem.
The poll() interface is made accessible from Ocaml in
Netsys_posix.
fork()
One of the main difference of the POSIX platforms compared with
Windows is that there is the fork() system call. Ocamlnet provides
multi-processing support in the netplex library (see Netplex_mp).
This makes it easy to manage a set of child processes that are used
as parallel running "workers" (e.g. for accepting network connections).
There is a registry of user functions to be run after a new child
process has been forked off (see
Netsys_posix.register_post_fork_handler). The intention is that the
child can close descriptors it would otherwise share with the parent.
One should note that there is a basic incompatibility between code
that uses fork() to achieve parallelism and multi-threading. The
problem is that the state of synchronization objects like mutexes is
undefined after fork(). In the C language, one can try to define
special handlers that run before/after the fork() to fix up such
objects. There is no such possibility in Ocaml (the mentioned
registry cannot be used for this purpose). One simply should not call
fork() while there is more than one thread. It is of course allowed
to start threads in the sub processes.
For the multi-processing capability of netplex this means that one
must not create threads in the master process (from which the worker
processes are forked off). There is no such restriction for the
children.
There is now extended support for starting subprograms. First, there
is Netsys_posix.spawn which provides a comfortable way for starting
programs. The interface is patterned after the posix_spawn() system
call although the implemention is not using posix_spawn() yet, but
a traditional fork/exec combination.
(In a later version of Ocamlnet it is planned to give the user the
choice of using posix_spawn() as the underlying system interface.
On some OS, posix_spawn() is highly optimized and significantly
faster than fork/exec.)
Second, it is now possible to let Ocamlnet watch for the termination
of child processes: Netsys_posix.watch_subprocess. This function
arranges that the SIGCHLD signal is caught, and that the process
termination is reported as a file descriptor event. By means of
careful programming this even works for multi-threaded applications
(where signal handling is notoriously difficult).
POSIX systems usually already provide RPC functionality, as some
deeply-integrated networking protocols like NFS are based on it.
Traditionally, there is a program called portmapper that acts as
registry of RPC services running on a machine. Ocamlnet can register
servers in portmapper, and it can also look up services there.
On some systems, there is a newer version of portmapper called
rpcbind. Fortunately, rpcbind is backward-compatible, and also
provides a portmapper-style interface.
There is some problem, though, on systems that also provide the XTI
networking API which is a replacement for the socket API. Whereas
TCP/IP can be used with both XTI and sockets, the local network
connections the API's provide are incompatible: A local XTI endpoint
cannot be contacted over the socket API, and a local socket endpoint
(i.e. a Unix domain socket) cannot be contacted over XTI. There is
the helper library Rpc_xti_client that allows to connect to a local
XTI server.
On Cygwin Ocamlnet should behave like on a POSIX machine.
General remark: The Win32 port is still very experimental, and far from being bug-free! Development progresses very slowly. Please don't expect wonders.
There is now a lot of support for Win32, even for asynchronous network programming. The minimum Windows version is Windows XP SP2. One should also mention that some functions are only supported when the Ocaml program is compiled as multi-threaded program, as sometimes helper threads are needed to emulate a certain behavior. For example, Win32 limits the number of file handles to 64 per thread when one watches them for events. Ocamlnet includes support to overcome this limit by creating helper threads as needed.
As a substitute for Unix Domain sockets, Ocamlnet provides access to
Win32 named pipes. These are, for security reasons, restricted to
local connection (within the same machine). As named pipes exist in a
special file system, one cannot create named pipes in arbitrary
directories (unlike Unix Domain sockets). As a workaround, it is also
supported to create named pipes with random names, and to write these
names into text files. A number of functions that used to expect Unix
Domain socket files as input can now also deal with these text files,
and will automatically map these files to the referenced named pipes
(see Netplex_sockserv.any_file_client_connector).
The RPC implementation supports TCP/IP sockets and named pipes, both for clients and for servers. As for POSIX, the clients and servers can be programmed in an asynchronous way.
Netplex is supported, but only for multi-threaded containers. The multi-processing containers are not available. Netplex can manage both TCP connections and named pipes.
The Shell library works, too. As for POSIX, it is possible to create complex pipelines between the started shell commands, and it is possible to read the output of and to provide input to the commands at the same time. All asynchronous command execution features are available! The only limitation is that one can redirect only stdin/stdout/stderr of the started commands, and not arbitrary descriptors as for POSIX (this is a restriction of Win32).
The Equeue library is supported. The GTK- and TclTk-specific extensions do not work, however. The SSL add-on is untested.
The Netshm library is only partially supported: The managed memory object must be file-backed and cannot (yet) live in RAM-only shared memory.
The Nethttpd library is fully supported.
XXX: Netcgi2
The missing Win32 calls are made available via Netsys_win32. The
accessible objects are:
Netsys_win32, pipe server
objects are emulated to simplify this for user code, so that pipe
servers look very much like socket servers.Netsys_win32 provides helper threads allowing one to
read from an arbitrary file handle in an asynchronous way, even if Win32
does not support that for the type of handle. This is intended for
reading from anonymous pipes (as returned by Unix.pipe).
A central idea here is the concept of "proxy descriptors". Many of the
mentioned objects have a complex inner structure - e.g. a pipe server
can mean an open number of Win32 file handles. However, it is intended
to make these objects look like in POSIX, and that means that there is
only a single descriptor referencing the object. The proxy descriptors
are additional file descriptors Ocamlnet allocates only for the purpose
of referencing these objects. For example, a pipe server is mainly
represented by the type Netsys_win32.w32_pipe_server, and there are
a lot of functions dealing with such values, e.g.
Netsys_win32.pipe_shutdown_server for shutting a server down.
Of course, a Netsys_win32.w32_pipe_server is not a file descriptor,
but a complex Ocaml record. The question is now how to pass such an
object to functions that only accept file descriptors as input? The
solution looks simple although the implementation causes a lot of
headache: A special file descriptor is allocated, and there is a global
table mapping these descriptors to the real objects. For instance, you
can get the proxy descriptor of a pipe server by calling
let (fd : Unix.file_descr) =
Netsys_win32.pipe_server_descr psrv
for a pipe server object psrv. You can pass fd around, and when
a called function requires the pipe server object again, one can
map fd back by:
let (psrv : Netsys_win32.w32_pipe_server) =
Netsys_win32.lookup_pipe_server fd
Like other descriptors, fd must be closed after use. The tricky part
of the implementation is that unreferenced proxy descriptors are
detected, and that the entry of the global mapping table is deleted
then.
As mentioned, Ocamlnet supports asynchronous I/O for the supported
Win32 objects. For some objects, Win32 has built-in support in the
form of so-called overlapped I/O. This kind of I/O is, unfortunately,
different than what POSIX provides (poll-style I/O). The analysis
shows that the main difference is that in Win32 one has to start an
I/O operation in order to asynchronously check whether and when it is
finally possible, whereas POSIX allows one to check the possibility of
I/O before one starts the operation. Ocamlnet hides this difference in
the Netsys_win32 wrapper by providing additional buffering. The
price is, however, that the generic read and write calls (or
better, ReadFile and WriteFile in Win32) can no longer be used.
As a substitute, Ocamlnet provides for each kind of object a special
set of read and write operations, e.g. Netsys_win32.pipe_read
for named pipes.
For simplicity of user code, there are also generalized read/write
operations: Netsys.gread and Netsys.gwrite. These work for
all descriptors supported by Unix.read and Unix.write, plus for
sockets, plus for all of the mentioned Win32-specific objects for
which reading and writing are reasonable I/O operations.
If overlapping I/O is unavailable for a kind of file handle, it is still possible to use the input and output threads in order to do asynchronous I/O for these handles. An example of such a handle type are anonymous pipes. The threads are automatically started and initiate synchronous I/O operations when needed. The completion of the operations is signalled via condition variables to the caller, so that it is possible to provide an asynchronous API to the file handle.
Generally, the functions in Netsys are available on all platforms
(but not necessarily in all variants), whereas Netsys_posix and
Netsys_win32 contain the platform-specific stuff.
In order to read and write file descriptors, one should first get
the kind of descriptor fd:
let st = Netsys.get_fd_style fd
Now, one can read and write to the file descriptor by calling
Netsys.gread and Netsys.gwrite (or by some of the provided
variants of these functions). These functions need st as input,
and internally call the right system function to do the requested
I/O operation.
There is also Netsys.gshutdown for shutting a descriptor down
(possibly only half-way), and Netsys.gclose for closing a
descriptor.
In order to wait for a single descriptor, one can call one of the
Netsys functions doing so, e.g. Netsys.wait_until_readable.
These functions block execution until the requested I/O operation
is possible, or until a timeout elapses.
If several descriptors need to be waited for, there is the
portable class type Netsys_pollset.pollset. One can add several
descriptors to pollsets and wait until I/O operations for one of the
descriptors become possible. There are implementations for both POSIX
and Win32. Netsys_pollset_generic.standard_pollset automatically
selects the best pollset implementation for "normal use".
For porting programs that are still written around Unix.select,
there is an emulation of select on top of pollsets:
Netsys_pollset_generic.select_emulation.
