Module Netmime_string

find_line_end s pos len: Searches the next line end (CR/LF or only LF), and returns the position. The search starts at position pos, and covers the next len bytes. Raises Not_found if there is no line end.

polymorphic version

find_line_start s pos len: Searches the next start, and returns its position. The line start is the position after the next line end (CR/LF or only LF). The search starts at position pos, and covers the next len bytes. Raises Not_found if there is no line end.

polymorphic version

find_double_line_start s pos len: Searches two adjacent line ends (each may be a CR/LF combination or a single LF), and returns the position after the second line end. The search starts at position pos, and covers the next len bytes. Raises Not_found if the mentioned pattern is not found.

polymorphic version

skip_line_ends s pos len: Skips over adjacent line ends (terminated by CR/LF or plain LF), and returns the position after the last line end. The search starts at position pos, and covers the next len bytes. Note that this function cannot raise Not_found.

polymorphic version

fold_lines_p f acc0 s pos len: Splits the substring of s from pos to pos+len into lines, and folds over these lines like List.fold_left. The function f is called as f acc p0 p1 p2 is_last where acc is the current accumulator (initialized with acc0), and

• p0 is the start position of the line in s
• p1 is the position of the line terminator in s
• p2 is the position after the line terminator in s
• is_last is true if this is the last line in the iteration

The lines can be terminated with CR/LF or LF. For the last line the terminator is optional (p1=p2 is possible).

The function is tail-recursive.

even more polymorphic

fold_lines f acc0 s pos len: Splits the substring of s from pos to pos+len into lines, and folds over these lines like List.fold_left. The function f is called as f acc line where acc is the current accumulator (initialized with acc0), and line is the current line w/o terminator.

The lines can be terminated with CR/LF or LF.

The function is tail-recursive.

Example: Get the lines as list:

         List.rev(fold_lines (fun l acc -> acc::l) [] s pos len)
      

even more polymorphic

iter_lines f s pos len: Splits the substring of s from pos to pos+len into lines, and calls f line for each line.

The lines can be terminated with CR/LF or LF.

even more polymorphic

skip_whitespace_left s pos len: Returns the smallest p with p >= pos && p < pos+len so that s.[p] is not a whitesapce character (space, TAB, CR, LF), and s.[q] is a whitespace character for all q<p. If this is not possible Not_found will be raised.

skip_whitespace_right s pos len: Returns the biggest p with p >= pos && p < pos+len so that s.[p] is not a whitesapce character (space, TAB, CR, LF), and s.[q] is a whitespace character for all q>p. If this is not possible Not_found will be raised.

fold_header f acc0 s pos len: Parses a MIME header in the string s from pos to exactly pos+len. The MIME header must be terminated by an empty line.

A folding operation is done over the header values while the lines are extracted from the string, very much like List.fold_left. For each header (n,v) where n is the name and v is the value, the function f is called as f acc n v.

If the header cannot be parsed, a Failure is raised.

Certain transformations may be applied (default: no transformations):

• If downcase is set, the header names are converted to lowercase characters
• If unfold is set, the line terminators are not included in the resulting values. This covers both the end of line characters at the very end of a header and the end of line characters introduced by continuation lines.
• If strip is set, preceding and trailing white space is removed from the value (including line terminators at the very end of the value)

list_header s pos len: Returns the headers as list of pairs (name,value).

For the meaning of the arguments see fold_header above.

let params, header_end_pos = scan_header s start_pos end_pos:

Deprecated.

Scans the mail header that begins at position start_pos in the string s and that must end somewhere before position end_pos. It is intended that in end_pos the character position following the end of the body of the MIME message is passed.

Returns the parameters of the header as (name,value) pairs (in params), and in header_end_pos the position of the character following directly after the header (i.e. after the blank line separating the header from the body).

• If downcase, header names are converted to lowercase characters
• The arguments unfold and strip have a slightly different meaning as for the new function fold_header above. In particular, whitespace is already stripped off the returned values if any of unfold or strip are enabled. (This is for backward compatibility.)

Also, this function is different because downcase and unfold are enabled by default, and only strip is not enabled.

The same for tagged strings

Polymorphic version

This function expects that the current position of the passed in_obj_stream is the first byte of the header. The function scans the header and returns it. After that, the stream position is after the header and the terminating empty line (i.e. at the beginning of the message body).

The options downcase, unfold, and strip have the same meaning as in scan_header.

Example

To read the mail message "file.txt":

 let ch = new Netchannels.input_channel (open_in "file.txt") in
 let stream = new Netstream.input_stream ch in
 let header = read_header stream in
 stream#close_in()  (* no need to close ch *)
 

This function writes the header to the passed out_obj_channel. The empty line following the header is also written.

Exact output format:

• The header is not folded, i.e. no additional CRLF sequences are inserted into the header to avoid long header lines. In order to produce correct headers, the necessary CRLF bytes must already exist in the field values. (You can use the function write_value below for this.)
• However, this function helps getting some details right. First, whitespace at the beginning of field values is suppressed. Example: write_header ch ["x","Field value"; "y"," Other value"] outputs:

   x: Field value\r\n
   y: Other value\r\n
   \r\n

• The end-of-line sequences LF, and CRLF, followed by whitespace are replaced by the passed soft_eol string. If the necessary space or tab character following the eol is missing, an additional space character will be inserted. Example: write_header ch ["x","Field\nvalue"; "y","Other\r\n\tvalue"] outputs:

   x: Field\r\n
    value
   y: Other\r\n
   \tvalue

• Empty lines (and lines only consisting of whitespace) are suppressed if they occur inside the header. Example: write_header ch ["x","Field\n\nvalue"] outputs:

   x: Field\r\n
    value

• Whitespace at the end of a header field is suppressed. One field is separated from the next field by printing eol once.

These rules ensure that the printed header will be well-formed with two exceptions:

• Long lines (> 72 characters) are neither folded nor rejected
• True 8 bit characters are neither properly encoded nor rejected

These two problems cannot be addressed without taking the syntax of the header fields into account. See below how to create proper header fields from s_token lists.

A token may be one of:

• QString s: The quoted string s, i.e a string between double quotes. Quoted pairs are already decoded in s.
• Control c: The control character c (0-31, 127, 128-255)
• Special c: The special character c, i.e. a character from the specials list
• DomainLiteral s: The bracketed string s, i.e. a string between brackets. Quoted pairs are already decoded in s.
• Comment: A string between parentheses. This kind of token is only generated when the option Return_comments is in effect.
• EncodedWord((charset,lang),encoding,encoded_word): An RFC-2047 style encoded word: charset is the name of the character set; lang is the language specifier (from RFC 2231) or ""; encoding is either "Q" or "B"; and encoded_word is the word encoded in charset and encoding. This kind of token is only generated when the option Recognize_encoded_words is in effect (if not, Atom is generated instead).
• Atom s: A string which is neither quoted not bracketed nor written in RFC 2047 notation, and which is not a control or special character, i.e. the "rest"
• End: The end of the string

An opaque type containing the information of s_token plus:

• where the token occurs
• RFC-2047 access functions

Return the s_token within the s_extended_token

Return the decoded word (the contents of the word after decoding the "Q" or "B" representation), and the character set of the decoded word (uppercase).

These functions not only work for EncodedWord. The function get_decoded_word returns for the other kinds of token:

• Atom: Returns the atom without decoding it
• QString: Returns the characters inside the double quotes, and ensures that any quoted pairs are decoded
• Control: Returns the one-character string
• Special: Returns the one-character string
• DomainLiteral: Returns the characters inside the brackets, and ensures that any quoted pairs are decoded
• Comment: Returns ""

The function get_charset returns "US-ASCII" for them.

Returns the language if the token is an EncodedWord, and "" for all other tokens.

Return the byte position where the token starts in the string (the first byte has position 0)

Return the line number where the token starts (numbering begins usually with 1)

Return the column of the line where the token starts (first column is number 0)

Return the length of the token in bytes

True iff the current token is white space (i.e. Special ' ', Special '\t', Special '\r' or Special '\n') and the last non-white space token was EncodedWord and the next non-white space token will be EncodedWord.

The background of this function is that white space between encoded words does not have a meaning, and must be ignored by any application interpreting encoded words.

The opaque type of a scanner for structured values

Creates a new mime_scanner scanning the passed string.

Return the current position, line, and column of a mime_scanner. The primary purpose of these functions is to simplify switching from one mime_scanner to another within a string:

 let scanner1 = create_mime_scanner ... s in
 ... now scanning some tokens from s using scanner1 ...
 let scanner2 = create_mime_scanner ... 
                  ?pos:(get_pos_of_scanner scanner1)
                  ?line:(get_line_of_scanner scanner1)
                  ?column:(get_column_of_scanner scanner1)
                  s in
 ... scanning more tokens from s using scanner2 ... 

Restriction: These functions are not available if the option Recognize_encoded_words is on. The reason is that this option enables look-ahead scanning; please use the location of the last scanned token instead.

Note: To improve the performance of switching, it is recommended to create scanner specs in advance (see the example my_scanner_spec above).

Returns the next token, or End if there is no more token. The token is returned both as extended and as normal token.

Returns all following tokens as a list (excluding End)

This function is included for backwards compatibility, and for all cases not requiring extended tokens.

It scans the passed string according to the list of special characters and the list of options, and returns the list of all tokens.

The sets of special characters defined by the RFCs 822 and 2045.

Scans a "text" value. The returned token list contains only Special, Atom and EncodedWord tokens. Spaces, TABs, CRs, LFs are returned (as Special) unless they occur between adjacent encoded words in which case they are suppressed. The characters '(', '[', and '"' are also returned as Special tokens, and are not interpreted as delimiters.

For instance, this function can be used to scan the "Subject" field of mail messages.

let name, params = scan_value_with_parameters s options: Scans values with annotations like name ; p1=v1 ; p2=v2 ; ... For example, MIME types like "text/plain;charset=ISO-8859-1" can be parsed.

The values may or may not be quoted. The characters ";", "=", and even "," are only accepted as part of values when they are quoted. On sytax errors, the function fails.

RFC 2231: This function supports some features of this RFC: Continued parameter values are concatenated. For example:

 Content-Type: message/external-body; access-type=URL;
    URL*0="ftp://";
    URL*1="cs.utk.edu/pub/moore/bulk-mailer/bulk-mailer.tar" 

This is returned as:

"message/external-body", 
   [ ("access-type", "URL");
     ("URL", "ftp://cs.utk.edu/pub/moore/bulk-mailer/bulk-mailer.tar") ]
      ) 

However, encoded parameter values are not handled specially. The parameter title*=us-ascii'en-us'This%20is%20%2A%2A%2Afun%2A%2A%2A would be returned as ("title*", "us-ascii'en-us'This%20is%20%2A%2A%2Afun%2A%2A%2A"). Use scan_values_with_parameters_ep instead (see below).

Raises Failure on syntax errors.

The type of encoded parameters (RFC 2231)

Return the decoded value of the parameter, the charset (uppercase), and the language. If the charset is not available, "" will be returned. If the language is not available, "" will be returned.

Creates a parameter from a value (in decoded form). The parameter may have a charset and a language.

Prints a parameter to the formatter (as toploop printer)

let name, params = scan_value_with_parameters_ep s options: This version of the scanner copes with encoded parameters according to RFC 2231. Note: "ep" means "encoded parameters".

Example: doc.html;title*=us-ascii'en-us'This%20is%20%2A%2A%2Afun%2A%2A%2A

The parameter title would be returned as:

• name is "title"
• value is "This is ***fun***"
• charset is "US-ASCII"
• language is "en-us"

Raises Failure on syntax errors.

let name, params = scan_mime_type s options: Scans MIME types like text/plain; charset=iso-8859-1 The name of the type and the names of the parameters are converted to lower case.

Raises Failure on syntax errors.

let name, params = scan_mime_type_ep s options: This version copes with RFC-2231-encoded parameters.

Raises Failure on syntax errors.

let (main_type, sub_type) = split_mime_type content_type: Splits the MIME type into main and sub type, for example split_mime_type "text/plain" = ("text", "plain") . The returned strings are always lowercase.

Raises Failure on syntax errors.

Raised when the hard limit of the line length is exceeded

Writes the list of s_token to the out_obj_channel. The value is optionally folded into several lines while writing, but this is off by default. To enable folding, pass both maxlen1 and maxlen: The maxlen1 parameter specifies the length of the first line to write, the maxlen parameter specifies the length of the other lines.

If enabled, folding tries to ensure that the value is written in several lines that are not longer as specified by maxlen1 and maxlen. The value is split into lines by inserting "folding space" at certain locations (which is usually a linefeed followed by a space character, see below). The following table specifies between which tokens folding may happen:

               +=========================================================+
 1st   \   2nd | Atom | QString | DLiteral | EncWord | Special | Spec ' '|
 ==============+======+=========+==========+=========+=========+=========+
          Atom | FS   |  FS     |   FS     |   FS    |    -    |    F    |
       QString | FS   |  FS     |   FS     |   FS    |    -    |    F    |
 DomainLiteral | FS   |  FS     |   FS     |   FS    |    -    |    F    |
   EncodedWord | FS   |  FS     |   FS     |   FS    |    -    |    F    |
       Special | -    |  -      |   -      |   -     |    -    |    F    |
   Special ' ' | -    |  -      |   -      |   -     |    -    |    -    |
 ==============+======+=========+==========+=========+=========+=========+

The table shows between which two types of tokens a space or a folding space is inserted:

• FS: folding space
• F: linefeed without extra space
• -: nothing can be inserted here

Folding space is "\n ", i.e. only LF, not CRLF is used as end-of-line character. The function write_header will convert these LF to CRLF if needed.

Special '\t' is handled like Special ' '. Control characters are just printed, without folding. Comments, however, are substituted by either space or folding space. The token End is ignored.

Furthermore, folding may also happen within tokens:

• Atom, Control, and Special are never split up into parts. They are simply printed.
• EncodedWords, however, are reformatted. This especially means: adjacent encoded words are first concatenated if possible (same character set, same encoding, same language), and then split up into several pieces with optimally chosen lengths. Note: Because this function gets s_token as input and not s_extended_token, it is not known whether Special ' ' tokens (or other whitespace) between adjacent EncodedWords must be ignored. Because of this, write_value only reformats adjacent encoded words when there is not any whitespace between them.
• QString may be split up in a special way unless fold_qstring is set to false. For example, "One Two Three" may be split up into three lines "One\n Two\n \ Three". Because some header fields explicitly forbid folding of quoted strings, it is possible to set ~fold_qstring:false (it is true by default). Note: Software should not rely on that the different types of whitespace (especially space and TAB) remain intact at the beginning of a line. Furthermore, it may also happen that additional whitespace is added at the end of a line by the transport layer.
• DomainLiteral: These are handled like QString. The parameter ~fold_literal:false turns folding off if it must be prevented, it is true by default.
• Comment: Comments are effectively omitted! Instead of Comment, a space or folding space is printed. However, you can output comments by passing sequences like Special "("; ...; Special ")" .

It is possible to get the actual number of characters back that can still be printed into the last line without making the line too long. Pass an int ref as unused to get this value (it may be negative!). Pass an int ref as hardunused to get the number of characters that may be printed until the hard limit is exceeded.

The function normally does not fail when a line becomes too long, i.e. it exceeds maxlen1 or maxlen. However, it is possible to specify a hard maximum length (hardmaxlen1 and hardmaxlen). If these are exceeded, the function will raise Line_too_long.

For electronic mail, a maxlen of 78 and a hardmaxlen of 998 is recommended.

Known Problems:

• The reformatter for EncodedWords takes into account that multi-byte characters must not be split up. However, this works only when the multi-byte character set is known to Netconversion. You can assume that UTF-8 and UTF-16 always work. If the character set is not known the reformatter may split the string at wrong positions.
• The reformatter for EncodedWords may parse the token, and if this fails, you will get the exception Malformed_code. This is only done in some special cases, however.
• The function prints spaces between adjacent atoms. Although this is allowed in principal, other MIME implementations might fail when there are spaces at unexpected locations. Workaround: If no spaces are desired, concatenate adjacent atoms before passing them to this function.

Further Tips:

• Pass ~maxlen1:0 and ~maxlen:0 to get shortest lines
• Use the reformatter for encoded words! It works well. For example, to output a long sentence, just wrap it into one EncodedWord. The reformatter takes care to fold the word into several lines.

Formats a parameter list. For example, [ "a", "b"; "c", "d" ] is transformed to the token sequence corresponding to ; a=b; c=d. If maxlen is specified, it is ensured that the individual parameter (e.g. "a=b;") is not longer than maxlen-1, such that it will fit into a line with maximum length maxlen. By default, no maximum length is guaranteed. If maxlen is passed, or if a parameter specifies a character set or language, the encoding of RFC 2231 will be applied. If these conditions are not met, the parameters will be encoded traditionally.

Splits a long URI according to the algorithm of RFC 2017. The input string must only contain 7 bit characters, and must be, if necessary, already be URL-encoded.

let [params1, value1; params2, value2; ...] = scan_multipart_body s start_pos end_pos boundary:

Scans the string s that is the body of a multipart message. The multipart message begins at position start_pos in s, and end_pos is the position of the character following the message. In boundary the boundary string must be passed (this is the "boundary" parameter of the multipart MIME type, e.g. multipart/mixed;boundary="some string" ).

The return value is the list of the parts, where each part is returned as pair (params, value). The left component params is the list of name/value pairs of the header of the part. The right component is the raw content of the part, i.e. if the part is encoded ("content-transfer-encoding"), the content is returned in the encoded representation. The caller is responsible for decoding the content.

The material before the first boundary and after the last boundary is not returned.

Multipart Messages

The MIME standard defines a way to group several message parts to a larger message (for E-Mails this technique is known as "attaching" files to messages); these are the so-called multipart messages. Such messages are recognized by the major type string "multipart", e.g. multipart/mixed or multipart/form-data. Multipart types MUST have a boundary parameter because boundaries are essential for the representation.

Multipart messages have a format like (where "_" denotes empty lines):

 ...Header...
 Content-type: multipart/xyz; boundary="abc"
 ...Header...
 _
 Body begins here ("prologue")
 --abc
 ...Header part 1...
 _
 ...Body part 1...
 --abc
 ...Header part 2...
 _
 ...Body part 2
 --abc
 ...
 --abc--
 Epilogue 

The parts are separated by boundary lines which begin with "--" and the string passed as boundary parameter. (Note that there may follow arbitrary text on boundary lines after "--abc".) The boundary is chosen such that it does not occur as prefix of any line of the inner parts of the message.

The parts are again MIME messages, with header and body. Note that it is explicitely allowed that the parts are even multipart messages.

The texts before the first boundary and after the last boundary are ignored.

Note that multipart messages as a whole MUST NOT be encoded. Only the PARTS of the messages may be encoded (if they are not multipart messages themselves).

Please read RFC 2046 if want to know the gory details of this brain-dead format.

Same as scan_multipart_body, but decodes the bodies of the parts if they are encoded using the methods "base64" or "quoted printable". Fails, if an unknown encoding is used.

scan_multipart_body_from_netstream s boundary create add stop:

Reads the MIME message from the netstream s block by block. The parts are delimited by the boundary.

Once a new part is detected and begins, the function create is called with the MIME header as argument. The result p of this function may be of any type.

For every chunk of the part that is being read, the function add is invoked: add p s k n.

Here, p is the value returned by the create invocation for the current part. s is the netstream. The current window of s contains the read chunk completely; the chunk begins at position k of the window (relative to the beginning of the window) and has a length of n bytes.

When the part has been fully read, the function stop is called with p as argument.

That means, for every part the following is executed:

• let p = create h
• add p s k1 n1
• add p s k2 n2
• ...
• add p s kN nN
• stop p

Important Precondition:

• The block size of the netstream s must be at least String.length boundary + 4

Exceptions:

• Exceptions can happen because of ill-formed input, and within the callbacks of the functions create, add, stop.
• If the exception happens while part p is being read, and the create function has already been called (successfully), the stop function is also called (you have the chance to close files). The exception is re-raised after stop returns.

This is the "next generation" multipart message parser. It is called as follows:

let parts = read_multipart_body f boundary s

As precondition, the current position of the stream s must be at the beginning of the message body. The string boundary must be the message boundary (without "--"). The function f is called for every message part, and the resulting list parts is the concatentation of the values returned by f.

The stream passed to f is a substream of s that begins at the first byte of the header of the message part. The function f can read data from the substream as necessary. The substream terminates at the end of the message part. This means that f can simply read the data of the substream from the beginning to the end. It is not necessary that f reads the substream until EOF, however.

After all parts have been read, the trailing material of stream s is skipped until EOF of s is reached.

Creates a boundary string that can be used to separate multipart messages. The string is 63 characters long and has the following "features":

• Most of the string consists of the minus character yielding a clear optical effect
• The string contains "=__". This sequence cannot be obtained by the quoted-printable encoding, so you need not to care whether strings encoded as quoted-printable contain the boundary.
• The string contains "<&>;" which is illegal in HTML, XML, and SGML.
• The string does not contain double quotes or backslashes, so you can safely put double quotes around it in the MIME header.
• The string contains nr, so you can safely distinguish between several boundaries occurring in the same MIME body if you assign different nr.
• The string contains a hash value composed of the first 256 bytes of all strings passed as random, and influenced by the current GC state.