Module Nethttp_client

module Nethttp_client: sig .. end

HTTP 1.1 client

Note for beginners: There is a simplified interface called Nethttp_client.Convenience.

Implements the following advanced features:

chunked message transport
persistent connections
connections in pipelining mode ("full duplex" connections)
modular authentication methods, currently Basic, Digest, and Negotiate
event-driven implementation; allows concurrent service for several network connections
HTTP proxy support, also with Basic and Digest authentication
SOCKS proxy support (1)
HTTPS support is now built-in, but requires that a TLS provider is initialized (see Tls for more information). HTTPS proxies are also supported (CONNECT method) (1)
Automatic and configurable retry of failed idempotent requests
Redirections can be followed automatically
Compressed message bodies can be automatically decoded (gzip only, method set_accept_encoding) (1)

Left out:

multipart messages, including multipart/byterange
content digests specified by RFC 2068 and 2069 (2)
content negotiation (2)
conditional and partial GET (2)
client-side caching (2)
HTTP/0.9 compatibility

(1) Since Ocamlnet-3.3

(2) These features can be implemented on top of this module if really needed, but there is no special support for them.

Related modules/software:

Nethttp_fs allows you to access HTTP servers in the style of filesystems
WebDAV: If you are looking for WebDAV there is an extension of this module: Webdav, which is separately available.

Thread safety

The module can be compiled such that it is thread-safe. In particular, one has to link the http_client_mt.cmxo object, and thread-safety is restricted to the following kinds of usage:

The golden rule is that threads must not share pipeline objects. If every thread uses its own pipeline, every thread will have its own set of state variables. It is not detected if two threads errornously share a pipeline, neither by an error message nor by implicit serialization. Strange things may happen.
The same applies to the other objects, e.g. http_call objects
The Convenience module even serializes; see below.

Types and Exceptions

exception Bad_message of string

The server sent a message which cannot be interpreted. The string indicates the reason.

exception No_reply

There was no response to the request because some other request failed earlier and it was not allowed to send the request again.

exception Too_many_redirections

While following redirections the limit has been reached

exception Name_resolution_error of string

Could not resolve this name - same as Uq_engines.Host_not_found

exception URL_syntax_error of string

This URL cannot be parsed after a redirection has been followed.

exception Timeout of string

A timeout. The string explains which connection is affected. New since Ocamlnet-3.3.

exception Proxy_error of int

An error status from a proxy. This is only used when extra proxy messages are used to configure the proxy (e.g. the CONNECT message).

exception Response_too_large

The length of the response exceeds the configured maximum

exception Http_protocol of exn

The request could not be processed because the exception condition was raised. The inner exception is one of the above defined.

exception Http_error of (int * string)

Deprecated in the scope of pipeline. The server sent an error message. The left component of the pair is the error code, the right component is the error text. This exception is only used by get_resp_body, and by the Nethttp_client.Convenience module. Note that for the latter usage the exception does not count as deprecated.

type status = [ `Client_error
       | `Http_protocol_error of exn
       | `Redirection
       | `Server_error
       | `Successful
       | `Unserved ]

Condensed status information of a HTTP call:

`Unserved: The call has not yet been finished
`HTTP_protocol_error e: An error on HTTP level occurred. Corresponds to the exception Http_protocol.
`Successful: The call is successful, and the response code is between 200 and 299.
`Redirection: The call is successful, and the response code is between 300 and 399.
`Client_error: The call failed with a response code between 400 and 499.
`Server_error: The call failed for any other reason.

type 'a auth_status = [ `Auth_error
       | `Continue of 'a
       | `Continue_reroute of 'a * int
       | `None
       | `OK
       | `Reroute of int ]

Status of HTTP-level authentication:

`None: Authentication wasn't tried.
`OK: The authentication protocol finished. What this means exactly depends on the protocol. For most protocols it just means that the server authenticated the client. For some protocols it may also mean that the client authenticated the server (mutual authentication).
`Auth_error: The authentication protocol did not succeed. Note that this state can also be reached for an otherwise successful HTTP response (i.e. code 200) when the client could not authenticate the server and the protocol demands this.
`Reroute trans_id: The request should be retried on a new connection for the transport identified by trans_id
`Continue: The authentication is still in progress. Normally the user should not see this state as the engine automatically continues the protocol. The argument of `Continue is private.
`Continue_reroute: the combination of continue and reroute: the auth protocol continues, but the next request must be sent on the indicated transport.

type 'message_class how_to_reconnect =

`\|`	`Send_again`	`(*`	Send the request automatically again	`*)`
`\|`	`Request_fails`	`(*`	Drop the request	`*)`
`\|`	`Inquire of ('message_class -> bool)`	`(*`	If the function return `true` send again, otherwise drop the request.	`*)`
`\|`	`Send_again_if_idem`	`(*`	Default behaviour: `Send_again` for idempotent methods (GET, HEAD), `Request_fails` for the rest	`*)`

How to deal with automatic reconnections, especially when the connection crashes.

type 'message_class how_to_redirect =

`\|`	`Redirect`	`(*`	Perform the redirection	`*)`
`\|`	`Do_not_redirect`	`(*`	No redirection	`*)`
`\|`	`Redirect_inquire of ('message_class -> bool)`	`(*`	If the function return `true` redirect, otherwise do not redirect. It is legal to set the `Location` header as part of the action performed by the function. (Should be an absolute http URL.)	`*)`
`\|`	`Redirect_if_idem`	`(*`	Default behaviour: `Redirect` for idempotent methods (GET, HEAD), `Do_not_redirect` for the rest	`*)`

type private_api

The private part of the http_call class type

type response_body_storage = [ `Body of unit -> Netmime.mime_body
       | `Device of unit -> Uq_io.out_device
       | `File of unit -> string
       | `Memory ]

How to create the response body:

`Memory: The response body is in-memory
`File f: The response body is stored into the file whose name is returned by f()
`Body f: The response body is stored into the object returned by f()
`Device f: The response is directly forwarded to the device obtained by f() (new since Ocamlnet-3.3)

When the function f is called in the latter cases the response header has already been received, and can be retrieved with the response_header method of the call object. Also, response_status_text, response_status_code, and response_status return meaningful values.

type synchronization =

`\|`	`Sync`	`(*`	The next request begins after the response of the last request has been received.	`*)`
`\|`	`Pipeline of int`	`(*`	The client is allowed to send several requests without waiting for responses. The number is the maximum number of unreplied requests that are allowed. A typical value: 5. If you increase this value, the risk becomes higher that requests must be repeatedly sent to the server in the case the connection crashes. Increasing is recommended if you send a bigger number of GET or HEAD requests to the server. Decreasing is recommended if you send large POST or PUT requests to the server. Values > 8 are interpreted as 8.	`*)`

This type determines whether to keep requests and responses synchronized or not.

The first request/response round is always done in Sync mode, because the protocol version of the other side is not known at that moment. Pipeline requires HTTP/1.1.

In previous versions of netclient there was a third option, Sync_with_handshake_before_request_body. This option is no longer necessary because the HTTP specification has been updated in the meantime, and there is a better mechanism now (the Expect header is set).

type resolver = Unixqueue.unix_event_system ->
       string -> (Unix.inet_addr option -> unit) -> unit

A name resolver is a function r called as r esys name reply. As name the name to resolve is passed. The resolver must finally call reply with either the resolved address or with None, indicating an error in the latter case. The event system esys can be used to carry out the resolution process in an asynchronous way, but this is optional.

Only 1:1 resolution is supported, 1:n resolution not.

type transport_layer_id = int

The ID identifies a requirement for the transport channel, especially whether plain HTTP is sufficient, or HTTPS needs to be used, and if so, whether there are further requirements for the TLS context. There are the predefined IDs:

Nethttp_client.http_trans_id for HTTP connections
Nethttp_client.https_trans_id for HTTPS connections without user certificates
Nethttp_client.proxy_only_trans_id for the restriction that this protocol can only be used via a web proxy

type http_options = {

`synchronization : synchronization;`	`(*`	Default: `Pipeline 5`.	`*)`
`maximum_connection_failures : int;`	`(*`	This option limits the number of connection attempts. Default: 2	`*)`
`maximum_message_errors : int;`	`(*`	This option limits the number of protocol errors tolerated per request. If a request leads to a protocol error, the connection is shut down, the server is connected again, and the request is tried again (if the kind of the message allows retransmission). If a request repeatedly fails, this option limits the number of retransmissions. Default: 2	`*)`
`inhibit_persistency : bool;`	`(*`	This option turns persistent connections off. Default: `false` It is normally not necessary to change this option.	`*)`
`connection_timeout : float;`	`(*`	If there is no network transmission for this period of time, the connection is shut down, and tried again. Default: 300.0 (seconds) It may be necessary to increase this value if HTTP is used for batch applications that contact extremely slow services.	`*)`
`number_of_parallel_connections : int;`	`(*`	The client keeps up to this number of parallel connections to a single content server or proxy. Default: 2 You may increase this value if you are mainly connected with an HTTP/1.0 proxy.	`*)`
`maximum_redirections : int;`	`(*`	The maximum number of redirections per message	`*)`
`handshake_timeout : float;`	`(*`	The timeout when waiting for "100 Continue". Default: 1.0	`*)`
`resolver : resolver;`	`(*`	The function for name resolution	`*)`
`configure_socket : Unix.file_descr -> unit;`	`(*`	A function to configure socket options	`*)`
`tls : Netsys_crypto_types.tls_config option;`	`(*`	The TLS configuration to use by default for https URLs. (This can be overridden per request by using a different `transport_layer_id`.)	`*)`
`schemes : (string * Neturl.url_syntax * int option * transport_layer_id) list;`	`(*`	The list of supported URL schemes. The tuples mean `(scheme, syntax, default_port, cb)`. By default, the schemes "http", "https", and "ipp" are supported.	`*)`
`verbose_status : bool;`
`verbose_request_header : bool;`
`verbose_response_header : bool;`
`verbose_request_contents : bool;`
`verbose_response_contents : bool;`
`verbose_connection : bool;`
`verbose_events : bool;`	`(*`	Enable various debugging message types. `verbose_status`: reports about status of received documents `verbose_request_header`: prints the header sent to the server `verbose_request_contents`: prints the document sent to the server `verbose_response_header`: prints the header of the answer from the server `verbose_response_contents`: prints the document received from the server `verbose_connection`: reports many connection events; authentication, too. `verbose_events`: everything about the interaction with `Unixqueue` By default, `verbose_status` and `verbose_connection` are enabled. Note that you also have to set `Debug.enable` to `true` to see any log message at all!	`*)`

}

Options for the whole pipeline. It is recommended to change options the following way:

    let opts = pipeline # get_options in
    let new_opts = { opts with <field> = <value>; ... } in
    pipeline # set_options new_opts

New fields can be added anytime to this record, and this style of changing options is transparent to field additions.

type header_kind = [ `Base | `Effective ]

The `Base header is set by the user of http_call and is never changed during processing the call. The `Effective header is a copy of the base header at the time the request is sent. The effective header contains additions like Content-length and authentication info.

class type http_call = object .. end

The container for HTTP calls

class type tls_cache = object .. end

A cache object for storing TLS session data

val null_tls_cache : unit -> tls_cache

This "cache" does not remember any session

val unlim_tls_cache : unit -> tls_cache

Returns a simple cache without limit for the number of cached sessions

class type transport_channel_type = object .. end

HTTP methods

class virtual generic_call : object .. end

This class is an implementation of http_call.

The following classes are implementations for the various HTTP methods. These classes do not initialize the call object.

class get_call : http_call

class trace_call : http_call

class options_call : http_call

class head_call : http_call

class post_call : http_call

class put_call : http_call

class delete_call : http_call

The following classes initialize the request message of the call (header and body). These classes are also backward compatible to the classes found in earlier versions of netclient.

class get : string -> http_call

Argument: URI

class trace : string -> int -> http_call

Arguments: URI, maximum number of hops

class options : string -> http_call

Argument: URI or "*"

class head : string -> http_call

Argument: URI

class post : string -> (string * string) list -> http_call

Arguments: URI, parameter list to be transferred as application/x-www-form-urlencoded body

class post_raw : string -> string -> http_call

Arguments: URI, body

class put : string -> string -> http_call

Arguments: URI, body

class delete : string -> http_call

Argument: URI

Authentication

class type key = object .. end

A key is a user/password combination for a certain realm

val key : user:string ->
       password:string ->
       realm:string -> domain:Neturl.url list -> key

Create a key object. user and password are encoded in UTF-8.

val key_creds : user:string ->
       creds:(string * string * (string * string) list) list ->
       http_options -> key

Create a key object from a list of credentials

class type key_handler = object .. end

class key_ring : ?uplink:#key_handler -> ?no_invalidation:bool -> unit -> object .. end

The key_ring is a cache for keys.

class type auth_session = object .. end

An auth_session represents an authenticated session

class type auth_handler = object .. end

An authentication handler has the capability of adding the necessary headers to messages.

class basic_auth_handler : ?enable_reauth:bool -> ?skip_challenge:bool -> #key_handler -> auth_handler

Basic authentication.

class digest_auth_handler : #key_handler -> auth_handler

Digest authentication.

class unified_auth_handler : ?insecure:bool -> #key_handler -> auth_handler

Support both digest and basic authentication, with preference to digest.

class generic_auth_handler : #key_handler -> (module Nethttp.HTTP_CLIENT_MECHANISM) list -> auth_handler

Authenticate with the passed generic HTTP mechanisms

For the Negotiate method (SPNEGO/GSSAPI), have a look at Netmech_spnego_http.

Transport

A connection cache is an object that keeps connections open that are currently unused. A connection cache can be shared by several pipelines.

type connection_cache = Nethttp_client_conncache.connection_cache

val close_connection_cache : connection_cache -> unit

Closes all descriptors known to the cache

val create_restrictive_cache : unit -> connection_cache

A restrictive cache closes connections as soon as there are no pending requests.

val create_aggressive_cache : unit -> connection_cache

This type of cache tries to keep connections as long open as possible. The consequence is that users are responsible for closing the descriptors (by calling close_connection_cache) when the cache is no longer in use.

val http_trans_id : transport_layer_id

Identifies the pure HTTP transport (without SSL), with or without web proxies

val https_trans_id : transport_layer_id

Identifies anonymous HTTPS transport (i.e. no client certificates), with or without web proxies.

val spnego_trans_id : transport_layer_id

Identifies an anonymous HTTPS transport that is additionally authenticated via SPNEGO (as described in RFC 4559)

val proxy_only_trans_id : transport_layer_id

Identifies web proxy connections. Use this to e.g. send an FTP URL to a web proxy via HTTP

val new_trans_id : unit -> transport_layer_id

Allocates and returns a new ID

val http_transport_channel_type : transport_channel_type

Transport via HTTP

val https_transport_channel_type : Netsys_crypto_types.tls_config -> transport_channel_type

Create a new transport for HTTPS and this configuration. As of OCamlnet-4, https is automatically enabled if Netsys_crypto.current_tls returns something, i.e. if TLS is globally initialized, so there is often no reason to configure a special https transport with this function. You still need it if you want to enable special configurations per request:

        let my_trans_id = Nethttp_client.new_trans_id()
        let my_tct = Nethttp_client.https_transport_channel_type my_tls_config
        pipeline # configure_transport my_trans_id my_tct;

Now you can enable this special configuration for a request object call:

        call # set_transport_layer my_trans_id

If you want to change the TLS configuration for the whole pipeline, just set the tls field of Nethttp_client.http_options.

type proxy_type = [ `Http_proxy | `Socks5 ]

Pipelines

class pipeline : object .. end

A pipeline is a queue of HTTP calls to perform

Example using the pipeline:

 let call = new get "http://server/path" in
 let pipeline = new pipeline in
 pipeline # add call;
 pipeline # run();    (* Now the HTTP client is working... *)
 match call # status with
 | `Successful -> ...
 | ...

Auxiliary pipeline functions

val parse_proxy_setting : insecure:bool -> string -> string * int * (string * string * bool) option

Parses the value of an environment variable like http_proxy, i.e. an HTTP URL. The argument is the URL. Returns (host,port,auth) where auth may include user name, password, and the insecure flag.

val parse_no_proxy : string -> string list

Parses the value of an environment variable like no_proxy. Returns the list of domains.

Convenience module for simple applications

Do open Nethttp_client.Convenience for simple applications.

module Convenience: sig .. end

Debugging

module Debug: sig .. end

This web site is published by Informatikbüro Gerd Stolpmann

Plasma	GitLab	Archive
Projects	Blog	Knowledge