Not everything works in the same way for all operating systems. This documents describes the different levels of support for the various platforms.

POSIX

Ocamlnet has been primarily developed for POSIX systems. Most libraries should work on all systems, but there are also some more special features that are designed for certain systems only.

Banning select()

On POSIX systems, the select() system call is not used anymore. As a general-purpose replacement Ocamlnet favors poll(). In future versions of Ocamlnet, there will also be support for the improved versions of poll() some systems provide, such as epoll on Linux, kqueue on BSD, and /dev/poll on Solaris. Ocamlnet is already prepared for this change.

The reason for banning select() are the limited capabilities of this API. Especially, it cannot handle file descriptors whose numeric values exceed a system-dependent limit. Although this limit is quite high (usually 1024) there are applications that need more descriptors. (Note that it is not sufficient to simply increase the maximum number of descriptors a process can have - the select() call is still restricted in the mentioned sense, and this seems to be unfixable.) The poll() interface does not have this problem.

The poll() interface is made accessible from Ocaml in Netsys_posix.

Multi-processing and fork()

One of the main difference of the POSIX platforms compared with Windows is that there is the fork() system call. Ocamlnet provides multi-processing support in the netplex library (see Netplex_mp). This makes it easy to manage a set of child processes that are used as parallel running "workers" (e.g. for accepting network connections).

There is a registry of user functions to be run after a new child process has been forked off (see Netsys_posix.register_post_fork_handler). The intention is that the child can close descriptors it would otherwise share with the parent.

One should note that there is a basic incompatibility between code that uses fork() to achieve parallelism and multi-threading. The problem is that the state of synchronization objects like mutexes is undefined after fork(). In the C language, one can try to define special handlers that run before/after the fork() to fix up such objects. There is no such possibility in Ocaml (the mentioned registry cannot be used for this purpose). One simply should not call fork() while there is more than one thread. It is of course allowed to start threads in the sub processes.

For the multi-processing capability of netplex this means that one must not create threads in the master process (from which the worker processes are forked off). There is no such restriction for the children.

Starting programs as sub processes

There is now extended support for starting subprograms. First, there is Netsys_posix.spawn which provides a comfortable way for starting programs. The interface is patterned after the posix_spawn() system call although the implemention is not using posix_spawn() yet, but a traditional fork/exec combination. (In a later version of Ocamlnet it is planned to give the user the choice of using posix_spawn() as the underlying system interface. On some OS, posix_spawn() is highly optimized and significantly faster than fork/exec.)

Second, it is now possible to let Ocamlnet watch for the termination of child processes: Netsys_posix.watch_subprocess. This function arranges that the SIGCHLD signal is caught, and that the process termination is reported as a file descriptor event. By means of careful programming this even works for multi-threaded applications (where signal handling is notoriously difficult).

RPC integration

POSIX systems usually already provide RPC functionality, as some deeply-integrated networking protocols like NFS are based on it. Traditionally, there is a program called portmapper that acts as registry of RPC services running on a machine. Ocamlnet can register servers in portmapper, and it can also look up services there.

On some systems, there is a newer version of portmapper called rpcbind. Fortunately, rpcbind is backward-compatible, and also provides a portmapper-style interface.

There is some problem, though, on systems that also provide the XTI networking API which is a replacement for the socket API. Whereas TCP/IP can be used with both XTI and sockets, the local network connections the API's provide are incompatible: A local XTI endpoint cannot be contacted over the socket API, and a local socket endpoint (i.e. a Unix domain socket) cannot be contacted over XTI. There is the helper library Rpc_xti_client that allows to connect to a local XTI server.

Windows

Cygwin

On Cygwin Ocamlnet should behave like on a POSIX machine.

Win32

General remark: The Win32 port is still very experimental, and far from being bug-free! Development progresses very slowly. Please don't expect wonders.

There is now a lot of support for Win32, even for asynchronous network programming. The minimum Windows version is Windows XP SP2. One should also mention that some functions are only supported when the Ocaml program is compiled as multi-threaded program, as sometimes helper threads are needed to emulate a certain behavior. For example, Win32 limits the number of file handles to 64 per thread when one watches them for events. Ocamlnet includes support to overcome this limit by creating helper threads as needed.

As a substitute for Unix Domain sockets, Ocamlnet provides access to Win32 named pipes. These are, for security reasons, restricted to local connection (within the same machine). As named pipes exist in a special file system, one cannot create named pipes in arbitrary directories (unlike Unix Domain sockets). As a workaround, it is also supported to create named pipes with random names, and to write these names into text files. A number of functions that used to expect Unix Domain socket files as input can now also deal with these text files, and will automatically map these files to the referenced named pipes (see Netplex_sockserv.any_file_client_connector).

The RPC implementation supports TCP/IP sockets and named pipes, both for clients and for servers. As for POSIX, the clients and servers can be programmed in an asynchronous way.

Netplex is supported, but only for multi-threaded containers. The multi-processing containers are not available. Netplex can manage both TCP connections and named pipes.

The Shell library works, too. As for POSIX, it is possible to create complex pipelines between the started shell commands, and it is possible to read the output of and to provide input to the commands at the same time. All asynchronous command execution features are available! The only limitation is that one can redirect only stdin/stdout/stderr of the started commands, and not arbitrary descriptors as for POSIX (this is a restriction of Win32).

The Equeue library is supported. The GTK- and TclTk-specific extensions do not work, however. The SSL add-on is untested.

The Netshm library is only partially supported: The managed memory object must be file-backed and cannot (yet) live in RAM-only shared memory.

The Nethttpd library is fully supported.

XXX: Netcgi2

Win32 low-level

The missing Win32 calls are made available via Netsys_win32. The accessible objects are:

• Events: Win32 event objects can signal a condition (and look like condition variables)
• Named pipes: Win32 named pipes are network connections between a client endpoint and a server endpoint. Ocamlnet only supports local connections, however.
• Pipe servers: The notion of "pipe servers" does not exist in Win32 as kernel objects. Win32 sees a pipe server as a set of endpoints one can connect to, and the user code has to manage this set. In Netsys_win32, pipe server objects are emulated to simplify this for user code, so that pipe servers look very much like socket servers.
• Input threads: Netsys_win32 provides helper threads allowing one to read from an arbitrary file handle in an asynchronous way, even if Win32 does not support that for the type of handle. This is intended for reading from anonymous pipes (as returned by Unix.pipe).
• Output threads: a similar kind of object writing to file handles
• Processes
• Consoles

A central idea here is the concept of "proxy descriptors". Many of the mentioned objects have a complex inner structure - e.g. a pipe server can mean an open number of Win32 file handles. However, it is intended to make these objects look like in POSIX, and that means that there is only a single descriptor referencing the object. The proxy descriptors are additional file descriptors Ocamlnet allocates only for the purpose of referencing these objects. For example, a pipe server is mainly represented by the type Netsys_win32.w32_pipe_server, and there are a lot of functions dealing with such values, e.g. Netsys_win32.pipe_shutdown_server for shutting a server down. Of course, a Netsys_win32.w32_pipe_server is not a file descriptor, but a complex Ocaml record. The question is now how to pass such an object to functions that only accept file descriptors as input? The solution looks simple although the implementation causes a lot of headache: A special file descriptor is allocated, and there is a global table mapping these descriptors to the real objects. For instance, you can get the proxy descriptor of a pipe server by calling

 let (fd : Unix.file_descr) =
     Netsys_win32.pipe_server_descr psrv 

for a pipe server object psrv. You can pass fd around, and when a called function requires the pipe server object again, one can map fd back by:

 let (psrv : Netsys_win32.w32_pipe_server) =
     Netsys_win32.lookup_pipe_server fd

Like other descriptors, fd must be closed after use. The tricky part of the implementation is that unreferenced proxy descriptors are detected, and that the entry of the global mapping table is deleted then.

As mentioned, Ocamlnet supports asynchronous I/O for the supported Win32 objects. For some objects, Win32 has built-in support in the form of so-called overlapped I/O. This kind of I/O is, unfortunately, different than what POSIX provides (poll-style I/O). The analysis shows that the main difference is that in Win32 one has to start an I/O operation in order to asynchronously check whether and when it is finally possible, whereas POSIX allows one to check the possibility of I/O before one starts the operation. Ocamlnet hides this difference in the Netsys_win32 wrapper by providing additional buffering. The price is, however, that the generic read and write calls (or better, ReadFile and WriteFile in Win32) can no longer be used. As a substitute, Ocamlnet provides for each kind of object a special set of read and write operations, e.g. Netsys_win32.pipe_read for named pipes.

For simplicity of user code, there are also generalized read/write operations: Netsys.gread and Netsys.gwrite. These work for all descriptors supported by Unix.read and Unix.write, plus for sockets, plus for all of the mentioned Win32-specific objects for which reading and writing are reasonable I/O operations.

If overlapping I/O is unavailable for a kind of file handle, it is still possible to use the input and output threads in order to do asynchronous I/O for these handles. An example of such a handle type are anonymous pipes. The threads are automatically started and initiate synchronous I/O operations when needed. The completion of the operations is signalled via condition variables to the caller, so that it is possible to provide an asynchronous API to the file handle.

Hints for portable programming

Generally, the functions in Netsys are available on all platforms (but not necessarily in all variants), whereas Netsys_posix and Netsys_win32 contain the platform-specific stuff.

In order to read and write file descriptors, one should first get the kind of descriptor fd:

 let st = Netsys.get_fd_style fd 

Now, one can read and write to the file descriptor by calling Netsys.gread and Netsys.gwrite (or by some of the provided variants of these functions). These functions need st as input, and internally call the right system function to do the requested I/O operation.

There is also Netsys.gshutdown for shutting a descriptor down (possibly only half-way), and Netsys.gclose for closing a descriptor.

In order to wait for a single descriptor, one can call one of the Netsys functions doing so, e.g. Netsys.wait_until_readable. These functions block execution until the requested I/O operation is possible, or until a timeout elapses.

If several descriptors need to be waited for, there is the portable class type Netsys_pollset.pollset. One can add several descriptors to pollsets and wait until I/O operations for one of the descriptors become possible. There are implementations for both POSIX and Win32. Netsys_pollset_generic.standard_pollset automatically selects the best pollset implementation for "normal use".

For porting programs that are still written around Unix.select, there is an emulation of select on top of pollsets: Netsys_pollset_generic.select_emulation.