Module Netconversion

module Netconversion: sig .. end

Preliminaries

A character set is a set of characters where every character is identified by a code point. An encoding is a way of representing characters from a set in byte strings. For example, the Unicode character set has more than 96000 characters, and the code points have values from 0 to 0x10ffff (not all code points are assigned yet). The UTF-8 encoding represents the code points by sequences of 1 to 4 bytes. There are also encodings that represent code points from several sets, e.g EUC-JP covers four sets.

Encodings are enumerated by the type encoding, and names follow the convention `Enc_*, e.g. `Enc_utf8. Character sets are enumerated by the type charset, and names follow the convention `Set_*, e.g. `Set_unicode.

This module deals mainly with encodings. It is important to know that the same character set may have several encodings. For example, the Unicode character set can be encoded as UTF-8 or UTF-16. For the 8 bit character sets, however, there is usually only one encoding, e.g `Set_iso88591 is always encoded as `Enc_iso88591.

In a single-byte encoding every code point is represented by one byte. This is what many programmers are accustomed at, and what the O'Caml language specially supports: A string is a sequence of chars, where char means an 8 bit quantity interpreted as character. For example, the following piece of code allocates a string of four chars, and assigns them individually:

 let s = String.create 4 in
 s.[0] <- 'G';
 s.[1] <- 'e';
 s.[2] <- 'r';
 s.[3] <- 'd';
 

In a multi-byte encoding there are code points that are represented by several bytes. As we still represent such text as string, the problem arises that a single char, actually a byte, often represents only a fraction of a full multi-byte character. There are two solutions:

• Give up the principle that text is represented by string. This is, for example, the approach chosen by Camomile, another O'Caml library dealing with Unicode. Instead, text is represented as int array. This way, the algorithms processing the text can remain the same.
• Give up the principle that individual characters can be directly accessed in a text. This is the primary way chosen by Ocamlnet. This means that there is not any longer the possibility to read or write the nth character of a text. One can, however, still compose texts by just concatenating the strings representing individual characters. Furthermore, it is possible to define a cursor for a text that moves sequentially along the text. The consequence is that programmers are restricted to sequential algorithms. Note that the majority of text processing falls into this class.

 let b = Buffer.create 80 in
 Buffer.add b (ustring_of_uchar `Enc_utf8 71);  (* 71 = code point of 'G' *)
 Buffer.add b (ustring_of_uchar `Enc_utf8 101); (* 101 = code point of 'e' *)
 Buffer.add b (ustring_of_uchar `Enc_utf8 114); (* 114 = code point of 'r' *)
 Buffer.add b (ustring_of_uchar `Enc_utf8 100); (* 100 = code point of 'd' *)
 let s = Buffer.contents b
 

It is important to always remember that a char is no longer a character but simply a byte. In many of the following explanations, we strictly distinguish between byte positions or byte counts, and character positions or character counts.

There a number of special effects that usually only occur in multi-byte encodings:

• Bad encodings: Not every byte sequence is legal. When scanning such text, the functions will raise the exception Malformed_code when they find illegal bytes.
• Unassigned code points: It may happen that a byte sequence is a correct representation for a code point, but that the code point is unassigned in the character set. When scanning, this is also covered by the exception Malformed_code. When converting from one encoding to another, it is also possible that the code point is only unassigned in the target character set. This case is usually handled by a substitution function subst, and if no such function is defined, by the exception Cannot_represent.
• Incomplete characters: The trailing bytes of a string may be the correct beginning of a byte sequence for a character, but not a complete sequence. Of course, if that string is the end of a text, this is just illegal, and also a case for Malformed_code. However, when text is processed chunk by chunk, this phenomenon may happen legally for all chunks but the last. For this reason, some of the functions below handle this case specially.
• Byte order marks: Some encodings have both big and little endian variants. A byte order mark at the beginning of the text declares which variant is actually used. This byte order mark is a declaration written like a character, but actually not a character.

Unicode

Netconversion is centred around Unicode. The conversion from one encoding to another works by finding the Unicode code point of the character to convert, and by representing the code point in the target encoding, even if neither encodings have to do with Unicode. Of course, this approach requires that all character sets handled by Netconversion are subsets of Unicode.

The supported range of Unicode code points: 0 to 0xd7ff, 0xe000 to 0xfffd, 0x10000 to 0x10ffff. All these code points can be represented in UTF-8 and UTF-16. Netconversion does not know which of the code points are assigned and which not, and because of this, it simply allows all code points of the mentioned ranges (but for other character sets, the necessary lookup tables exist).

UTF-8: The UTF-8 representation can have one to four bytes. Malformed byte sequences are always rejected, even those that want to cheat the reader like "0xc0 0x80" for the code point 0. There is special support for the Java variant of UTF-8 (`Enc_java). `Enc_utf8 strings must not have a byte order mark (it would be interpreted as "zero-width space" character). However, the Unicode standard allows byte order marks at the very beginning of texts; use `Enc_utf8_opt_bom in this case.

UTF-16: When reading from a string encoded as `Enc_utf16, a byte order mark is expected at the beginning. The detected variant (`Enc_utf16_le or `Enc_utf16_be) is usually returned by the parsing function. The byte order mark is not included into the output string. - Some functions of this module cannot cope with `Enc_utf16 (i.e. UTF-16 without endianess annotation), and will fail.

Once the endianess is determined, the code point 0xfeff is no longer interpreted as byte order mark, but as "zero-width non-breakable space".

Some code points are represented by pairs of 16 bit values, these are the so-called "surrogate pairs". They can only occur in UTF-16.

UTF-32: This is very much the same as for UTF-16. There is a little endian version `Enc_utf32_le and a big endian version `Enc_utf32_be.

Subsets of Unicode

The non-Unicode character sets are subsets of Unicode. Here, it may happen that a Unicode code point does not have a corresponding code point. In this case, certain rules are applied to handle this (see below). It is, however, ensured that every non-Unicode code point has a corresponding Unicode code point. (In other words, character sets cannot be supported for which this property does not hold.)

It is even possible to create further subsets artificially. The encoding `Enc_subset(e,def) means to derive a new encoding from the existing one e, but to only accept the code points for which the definition function def yields the value true. For example, the encoding

 `Enc_subset(`Enc_usascii, 
             fun i -> i <> 34 && i <> 38 && i <> 60 && i <> 62) 

If a code point is not defined by the encoding but found in a text, the reader will raise the exception Malformed_code. When text is output, however, the subst function will be called for undefined code points (which raises Cannot_represent by default). The subst function is an optional argument of many conversion functions that allows it to insert a substitution text for undefined code points. Note, however, that the substitution text is restricted to at most 50 characters (because unlimited length would lead to difficult problems we would like to avoid).

Linking this module

Many encodings require lookup tables. The following encodings are built-in and always supported:

• Unicode: `Enc_utf8, `Enc_java, `Enc_utf16, `Enc_utf16_le, `Enc_utf16_be, `Enc_utf32, `Enc_utf32_le, `Enc_utf32_be
• Other: `Enc_usascii, `Enc_iso88591, `Enc_empty

Supported Encodings, Restrictions

I took the mappings from www.unicode.org, and the standard names of the character sets from IANA. Obviously, many character sets are missing that can be supported; especially ISO646 character sets, and many EBCDIC code pages. Stateful encodings like generic ISO-2022 have been omitted (stateless subsets of ISO-2022 like EUC can be supported, however; currently we support EUC-JP and EUC-KR).

Because of the copyright statement from Unicode, I cannot put the source tables that describe the mappings into the distribution. They are publicly available from www.unicode.org.

Known Problems

• The following charsets do not have a bijective mapping to Unicode: adobe_standard_encoding, adobe_symbol_encoding, adobe_zapf_dingbats_encoding, cp1002 (0xFEBE). The current implementation simply removes one of the conflicting code point pairs - this might not what you want.
• Japanese encodings: JIS X 0208: The character 1/32 is mapped to 0xFF3C, and not to 0x005C.

Interface

Naming conventions:

As it is possible to refer to substrings by either giving a byte offset or by counting whole characters, these naming conventions are helpful:

• Labels called range_pos and range_len refer to byte positions of characters, or substrings
• Labels called count refer to positions given as the number of characters relative to an origin

• A uchar is a single Unicode code point represented as int
• A ustring is a string of encoded characters
• A uarray is an array of int representing a string

exception Malformed_code

exception Cannot_represent of int

type encoding = [ `Enc_adobe_standard_encoding
       | `Enc_adobe_symbol_encoding
       | `Enc_adobe_zapf_dingbats_encoding
       | `Enc_asn1_T61
       | `Enc_asn1_iso646
       | `Enc_asn1_printable
       | `Enc_cp037
       | `Enc_cp1006
       | `Enc_cp1026
       | `Enc_cp1047
       | `Enc_cp424
       | `Enc_cp437
       | `Enc_cp500
       | `Enc_cp737
       | `Enc_cp775
       | `Enc_cp850
       | `Enc_cp852
       | `Enc_cp855
       | `Enc_cp856
       | `Enc_cp857
       | `Enc_cp860
       | `Enc_cp861
       | `Enc_cp862
       | `Enc_cp863
       | `Enc_cp864
       | `Enc_cp865
       | `Enc_cp866
       | `Enc_cp869
       | `Enc_cp874
       | `Enc_cp875
       | `Enc_empty
       | `Enc_eucjp
       | `Enc_euckr
       | `Enc_iso88591
       | `Enc_iso885910
       | `Enc_iso885911
       | `Enc_iso885913
       | `Enc_iso885914
       | `Enc_iso885915
       | `Enc_iso885916
       | `Enc_iso88592
       | `Enc_iso88593
       | `Enc_iso88594
       | `Enc_iso88595
       | `Enc_iso88596
       | `Enc_iso88597
       | `Enc_iso88598
       | `Enc_iso88599
       | `Enc_java
       | `Enc_jis0201
       | `Enc_koi8r
       | `Enc_macroman
       | `Enc_subset of encoding * (int -> bool)
       | `Enc_usascii
       | `Enc_utf16
       | `Enc_utf16_be
       | `Enc_utf16_le
       | `Enc_utf32
       | `Enc_utf32_be
       | `Enc_utf32_le
       | `Enc_utf8
       | `Enc_utf8_opt_bom
       | `Enc_windows1250
       | `Enc_windows1251
       | `Enc_windows1252
       | `Enc_windows1253
       | `Enc_windows1254
       | `Enc_windows1255
       | `Enc_windows1256
       | `Enc_windows1257
       | `Enc_windows1258 ] 

type charset = [ `Set_adobe_standard_encoding
       | `Set_adobe_symbol_encoding
       | `Set_adobe_zapf_dingbats_encoding
       | `Set_asn1_T61
       | `Set_asn1_iso646
       | `Set_asn1_printable
       | `Set_cp037
       | `Set_cp1006
       | `Set_cp1026
       | `Set_cp1047
       | `Set_cp424
       | `Set_cp437
       | `Set_cp500
       | `Set_cp737
       | `Set_cp775
       | `Set_cp850
       | `Set_cp852
       | `Set_cp855
       | `Set_cp856
       | `Set_cp857
       | `Set_cp860
       | `Set_cp861
       | `Set_cp862
       | `Set_cp863
       | `Set_cp864
       | `Set_cp865
       | `Set_cp866
       | `Set_cp869
       | `Set_cp874
       | `Set_cp875
       | `Set_iso88591
       | `Set_iso885910
       | `Set_iso885911
       | `Set_iso885913
       | `Set_iso885914
       | `Set_iso885915
       | `Set_iso885916
       | `Set_iso88592
       | `Set_iso88593
       | `Set_iso88594
       | `Set_iso88595
       | `Set_iso88596
       | `Set_iso88597
       | `Set_iso88598
       | `Set_iso88599
       | `Set_jis0201
       | `Set_jis0208
       | `Set_jis0212
       | `Set_koi8r
       | `Set_ks1001
       | `Set_macroman
       | `Set_unicode
       | `Set_usascii
       | `Set_windows1250
       | `Set_windows1251
       | `Set_windows1252
       | `Set_windows1253
       | `Set_windows1254
       | `Set_windows1255
       | `Set_windows1256
       | `Set_windows1257
       | `Set_windows1258 ] 

A number of the following functions can be made run faster if they are called several times for the same encoding. In this case, it is recommended to apply the function once partially with the encoding argument, and to call the resulting closure instead. For example, ustring_of_uchar supports this technique:

   let my_ustring_of_uchar = ustring_of_uchar my_enc in
   let s1 = my_ustring_of_uchar u1 ...
   let s2 = my_ustring_of_uchar u2 ... 

This is much faster than

   let s1 = ustring_of_uchar my_enc u1 ...
   let s2 = ustring_of_uchar my_enc u2 ... 

The availability of this optimization is indicated by the predicate PRE_EVAL(arg) where arg identifies the encoding argument.

Inlining

When a function can be inlined across module/library boundaries, this is indicated by the predicate INLINED. Of course, this works only for the ocamlopt compiler.

val encoding_of_string : string -> encoding

val string_of_encoding : encoding -> string

val is_ascii_compatible : encoding -> bool

val is_single_byte : encoding -> bool

val same_encoding : encoding -> encoding -> bool

val byte_order_mark : encoding -> string

val makechar : encoding -> int -> string

val ustring_of_uchar : encoding -> int -> string

val to_unicode : charset -> int -> int

val from_unicode : charset -> int -> int

val available_input_encodings : unit -> encoding list

val available_output_encodings : unit -> encoding list

val user_encoding : unit -> encoding option

val win32_code_pages : (int * encoding) list

Direct Conversion

 let s_utf8 = 
    convert ~in_enc:`Enc_iso88591 ~out_enc:`Enc_utf8 s_latin1 

which converts the ISO-8859-1 string s_latin1 to the UTF-8 string s_utf8.

It is also possible to convert while reading from or writing to a file. This use case is effectively handled by the class Netconversion.conversion_pipe. See the explanations of this class for examples.

val convert : ?subst:(int -> string) ->
       in_enc:encoding ->
       out_enc:encoding ->
       ?range_pos:int -> ?range_len:int -> string -> string

val recode_string : in_enc:encoding ->
       out_enc:encoding -> ?subst:(int -> string) -> string -> string

val recode : in_enc:encoding ->
       in_buf:string ->
       in_pos:int ->
       in_len:int ->
       out_enc:encoding ->
       out_buf:string ->
       out_pos:int ->
       out_len:int ->
       max_chars:int -> subst:(int -> string) -> int * int * encoding

class conversion_pipe : ?subst:int -> string -> in_enc:encoding -> out_enc:encoding -> unit -> Netchannels.io_obj_channel

class recoding_pipe : ?subst:int -> string -> in_enc:encoding -> out_enc:encoding -> unit -> Netchannels.io_obj_channel

Reading Text Using Cursors

A cursor is a reference to a character in an encoded string. The properties of the current character can be obtained, and the cursor can be moved relative to its current position.

For example, the following loop outputs the Unicode code points of all characters of the UTF-8 input string s:

 let cs = create_cursor `Enc_utf8 s in
 while not (cursor_at_end cs) do
   let n = cursor_char_count cs in
   let ch = uchar_at cs in
   printf "At position %d: %d\n" n ch;
   move cs;
 done
 

For a more exact definition, cursors are modeled as follows: The reference to the encoded string is contained in the cursor. This can be a complete string, or an arbitrary substring (denoted by a range of valid byte positions). The cursor position can be initially set to an arbitrary byte position of the encoded string.

Cursor positions can be denoted by

• byte positions p in the encoded string, or by
• character counts n relative to the initial position.

• n=0: This is always the initial cursor position
• n>0: Positive char counts refer to characters right to the initial character. The rightmost position is the position n_max past the rightmost character. The rightmost position does not have a code point.
• n<0: Negative char counts refer to characters left to the initial character. The leftmost position is the position n_min of the leftmost character.

Cursors are moved to the left or right of their current position by a whole number of characters. When it is tried to move them past the leftmost or rightmost position, the cursor is placed to the leftmost or rightmost position, respectively, and the exception Cursor_out_of_range is raised.

There are two cases of illegal encodings:

• When the last byte sequence of the encoded string is an incomplete multi-byte character, this is detected, and the special exception Partial_character is raised when the code point of this character is read. Note that this can only happen at position n_max-1. It is allowed to move beyond this character to n_max.
• When an illegal byte sequence occurs in the encoded string (including an incomplete multi-byte character at the beginning of the string), it is not possible to move the cursor to this character, or across this character. When it is tried to do so, the cursor stops just before the bad sequence, and the exception Malformed_code is raised.

type cursor 

exception End_of_string

exception Cursor_out_of_range

exception Partial_character

exception Byte_order_mark

val create_cursor : ?range_pos:int ->
       ?range_len:int ->
       ?initial_rel_pos:int ->
       encoding -> string -> cursor

val reinit_cursor : ?range_pos:int ->
       ?range_len:int ->
       ?initial_rel_pos:int ->
       ?enc:encoding -> string -> cursor -> unit

val copy_cursor : ?enc:encoding -> cursor -> cursor

val cursor_target : cursor -> string

val cursor_range : cursor -> int * int

val cursor_initial_rel_pos : cursor -> int

val cursor_char_count : cursor -> int

val cursor_pos : cursor -> int

val uchar_at : cursor -> int

val cursor_byte_length : cursor -> int

val cursor_at_end : cursor -> bool

val move : ?num:int -> cursor -> unit

val cursor_encoding : cursor -> encoding

val cursor_blit : cursor -> int array -> int -> int -> int

val cursor_blit_maxlen : cursor -> int

val cursor_blit_positions : cursor -> int array -> int -> int -> int

let n1 = cursor_blit     cs ua ua_pos len in
 let n2 = cursor_blit_pos cs pa pa_pos len in
 assert (n1 = n2)

Byte Order Marks

Because UTF-16 allows both little and big endian, files and other permanent representations of UTF-16 text are usually prepended by a byte order mark (BOM). There is confusion about the BOM among Unicode users, so the following explanations may be helpful.

Of course, the BOM is only used for external representations like files, as the endianess is always known for in-memory representations by the running program. This module has six encoding identifiers:

• `Enc_utf16: UTF-16 where the endianess is unknown
• `Enc_utf16_le: UTF-16 little endian
• `Enc_utf16_be: UTF-16 big endian
• `Enc_utf32: UTF-32 where the endianess is unknown
• `Enc_utf32_le: UTF-32 little endian
• `Enc_utf32_be: UTF-32 big endian

There is one problem, though. Unfortunately, the code point 0xfeff is also used for the normal "zero width non-breakable space" character. When this code point occurs later in the text, it is interpreted as this character. Of course, this means that one must know whether there is a BOM at the beginning, and if not, one must know the endianess. One cannot program in the style "well, let's see what is coming and guess".

Unicode also allows a BOM for UTF-8 although it is meaningless to specify the endianess. If you create the cursor with the encoding `Enc_utf8 nothing is done about this, and you get the BOM as normal character. If you create the cursor with `Enc_utf8_opt_bom, the BOM is treated specially like in the UTF-16 and -32 cases (with the only difference that it is optional for UTF-8).

The functions of this module can all deal with BOMs when reading encoded text. In most cases, the BOM is hidden from the caller, and just handled automatically. Cursors, however, treat BOMs as special characters outside of the code set (exception Byte_order_mark is raised). The writing functions of this module do not generate BOMs, however, as there is no way to tell them that a BOM is needed. The function byte_order_mark can be used to output the BOM manually.

Examples for Cursors

Create the cursor:

let cs = create_cursor `Enc_utf8 "B\195\164r";;

The cursor is now positioned at the 'B':

uchar_at cs returns 66 (i.e. B)

Move the cursor one character to the right. In UTF-8, this is a two-byte character consisting of the bytes 195 and 164:

move cs ;;

uchar_at cs returns 228 (i.e. a-Umlaut)

One can easily move the cursor to the end of the string:

move ~num:max_int cs ;;

This raises Cursor_out_of_range, but places the cursor at the end. This is the position past the last letter 'r':

uchar_at cs raises End_of_string

Go one character to the left:

move ~num:(-1) cs ;;

uchar_at cs returns 114 (i.e. r)

Cursors can only move relative to their current position. Of course, one can easily write a function that moves to an absolute position, like

 let move_abs n cs = 
    let delta = n - cursor_pos cs in
    move ~num:delta cs 

However, this operation is expensive (O(string length)), and should be avoided for efficient algorithms. Cursors are not arrays, and an algorithm should only be based on cursors when it is possible to iterate over the characters of the string one after another.

Unicode String Functions

val ustring_length : encoding -> ?range_pos:int -> ?range_len:int -> string -> int

val ustring_iter : encoding ->
       (int -> unit) -> ?range_pos:int -> ?range_len:int -> string -> unit

val ustring_map : encoding ->
       (int -> int list) -> ?range_pos:int -> ?range_len:int -> string -> string

val ustring_to_lower : encoding ->
       ?range_pos:int -> ?range_len:int -> string -> string

val ustring_to_upper : encoding ->
       ?range_pos:int -> ?range_len:int -> string -> string

val ustring_to_title : encoding ->
       ?range_pos:int -> ?range_len:int -> string -> string

val ustring_sub : encoding ->
       int -> int -> ?range_pos:int -> ?range_len:int -> string -> string

val ustring_compare : encoding ->
       (int -> int -> int) ->
       ?range_pos:int ->
       ?range_len:int -> string -> ?range_pos:int -> ?range_len:int -> string -> int

val code_cmp : int -> int -> int

val ci_code_cmp : int -> int -> int

val uarray_of_ustring : encoding ->
       ?range_pos:int -> ?range_len:int -> string -> int array

val ustring_of_uarray : ?subst:(int -> string) ->
       encoding -> ?pos:int -> ?len:int -> int array -> string

exception Malformed_code_at of int

val verify : encoding -> ?range_pos:int -> ?range_len:int -> string -> unit