Plasma GitLab Archive
Projects Blog Knowledge

Example_readme



The readme processor

The task of the readme processor is to convert a document conforming to the XML DTD "readme.dtd" into an HTML document or a text document. This example especially demonstrates how to use node extensions to add custom methods to nodes (see Intro_extensions), and how to use the object-oriented feature of late binding so that every node type behaves differently.

Note that the converter code dates back from 1999. Nowadays I would probably have written it as a purely functional transformer. This task is now left to the reader...

The readme DTD

The motivation for readme was that I often wrote two versions of files such as README and INSTALL explaining aspects of a distributed software archive; one version was ASCII-formatted, the other was written in HTML. Maintaining both versions meant double amount of work, and changes of one version could be forgotten in the other version. To improve this situation I invented the readme DTD which allows me to maintain only one source written as XML document, and to generate both the ASCII and the HTML version from it

In this section, I explain only the DTD. The readme DTD is contained in the PXP distribution together with the two converters to produce ASCII and HTML.

The documents have a simple structure: There are up to three levels of nested sections, paragraphs, item lists, footnotes, hyperlinks, and text emphasis. The outermost element has usually the type readme, it is declared by

<!ELEMENT readme (sect1+)>
<!ATTLIST readme
          title CDATA #REQUIRED>

This means that this element contains one or more sections of the first level (element type sect1), and that the element has a required attribute title containing character data (CDATA). Note that readme elements must not contain text data.

The three levels of sections are declared as follows:

<!ELEMENT sect1 (title,(sect2|p|ul)+)>

<!ELEMENT sect2 (title,(sect3|p|ul)+)>

<!ELEMENT sect3 (title,(p|ul)+)>

Every section has a title element as first subelement. After the title an arbitrary but non-empty sequence of inner sections, paragraphs and item lists follows. Note that the inner sections must belong to the next higher section level; sect3 elements must not contain inner sections because there is no next higher level.

Obviously, all three declarations allow paragraphs (p) and item lists (ul). The definition can be simplified at this point by using a parameter entity:

<!ENTITY % p.like "p|ul">

<!ELEMENT sect1 (title,(sect2|%p.like;)+)>

<!ELEMENT sect2 (title,(sect3|%p.like;)+)>

<!ELEMENT sect3 (title,(%p.like;)+)>

Here, the entity p.like is nothing but a macro abbreviating the same sequence of declarations; if new elements on the same level as p and ul are later added, it is sufficient only to change the entity definition. Note that there are some restrictions on the usage of entities in this context; most important, entities containing a left parenthesis must also contain the corresponding right parenthesis.

Note that the entity p.like is a parameter entity, i.e. the ENTITY declaration contains a percent sign, and the entity is referred to by %p.like;. This kind of entity must be used to abbreviate parts of the DTD; the general entities declared without percent sign and referred to as &amp;name; are not allowed in this context.

The title element specifies the title of the section in which it occurs. The title is given as character data, optionally interspersed with line breaks (br):

<!ELEMENT title (#PCDATA|br)*>

Compared with the title attribute of the readme element, this element allows inner markup (i.e. br) while attribute values do not: It is an error if an attribute value contains the left angle bracket < literally such that it is impossible to include inner elements.

The paragraph element p has a structure similar to title, but it allows more inner elements:

<!ENTITY % text "br|code|em|footnote|a">

<!ELEMENT p (#PCDATA|%text;)*>

Line breaks do not have inner structure, so they are declared as being empty:

<!ELEMENT br EMPTY>

This means that really nothing is allowed within br; you must always write <br></br> or abbreviated <br/>.

Code samples should be marked up by the code tag; emphasized text can be indicated by em:

<!ELEMENT code (#PCDATA)>

<!ELEMENT em (#PCDATA|%text;)*>

That code elements are not allowed to contain further markup while em elements do is a design decision by the author of the DTD.

Unordered lists simply consists of one or more list items, and a list item may contain paragraph-level material:

<!ELEMENT ul (li+)>

<!ELEMENT li (%p.like;)*>

Footnotes are described by the text of the note; this text may contain text-level markup. There is no mechanism to describe the numbering scheme of footnotes, or to specify how footnote references are printed.

<!ELEMENT footnote (#PCDATA|%text;)*>

Hyperlinks are written as in HTML. The anchor tag contains the text describing where the link points to, and the href attribute is the pointer (as URL). There is no way to describe locations of "hash marks". If the link refers to another readme document, the attribute readmeref should be used instead of href. The reason is that the converted document has usually a different system identifier (file name), and the link to a converted document must be converted, too.

<!ELEMENT a (#PCDATA)*>
<!ATTLIST a 
          href      CDATA #IMPLIED
          readmeref CDATA #IMPLIED
>

Note that although it is only sensible to specify one of the two attributes, the DTD has no means to express this restriction.

So far the DTD. Finally, here is a sample document for it:

<?xml version="1.0" encoding="ISO-8859-1"?>
<!DOCTYPE readme SYSTEM "readme.dtd">
<readme title="How to use the readme converters">
  <sect1>
    <title>Usage</title>
    <p>
      The <em>readme</em> converter is invoked on the command line by:
    </p>
    <p>
      <code>readme [ -text | -html ] input.xml</code>
    </p>
    <p>
      Here is a list of options:
    </p>
    <ul>
      <li>
        <p>
          <code>-text</code>: 
          specifies that ASCII output should be produced</p>
      </li>
      <li>
        <p>
           <code>-html</code>: 
           specifies that HTML output should be produced</p>
      </li>
    </ul>
    <p>
      The input file must be given on the command line. The converted output is
      printed to <em>stdout</em>.
    </p>
  </sect1>
  <sect1>
    <title>Author</title>
    <p>
      The program has been written by
      <a href="mailto:gerd@gerd-stolpmann.de">Gerd Stolpmann</a>.
    </p>
  </sect1>
</readme>

The readme converter to HTML

The converter from readme documents to HTML documents follows strictly the approach to define one extension class per element type. The generated HTML code is structurally similar to the readme source, because of this most elements can be converted in the following straigh-forward way: Given the input element

<e>content</e>

the conversion text is the concatenation of a computed prefix, the recursively converted content, and a computed suffix.

Only one element type cannot be handled by this scheme: footnote. Footnotes are collected while they are found in the input text, and they are printed after the main text has been converted and printed.

Now we comment to source code of the to_html.ml converter.

Header

open Pxp_types
open Pxp_document
open Pxp_dtd.Entity

Type declarations

class type footnote_printer =
  object
    method footnote_to_html : store_type -> out_channel -> unit
  end

and store_type =
  object
    method alloc_footnote : footnote_printer -> int
    method print_footnotes : out_channel -> unit
  end

For comments see the implementations below.

The class store of type store_type

The store is a container for footnotes. You can add a footnote by invoking alloc_footnote; the argument is an object of the class footnote_printer, the method returns the number of the footnote. The interesting property of a footnote is that it can be converted to HTML, so a footnote_printer is an object with a method footnote_to_html. The class footnote which is defined below has a compatible method footnote_to_html such that objects created from it can be used as footnote_printers.

The other method, print_footnotes prints the footnotes as definition list, and is typically invoked after the main material of the page has already been printed. Every item of the list is printed by footnote_to_html.

class store =
  object (self)

    val mutable footnotes = ( [] : (int * footnote_printer) list )
    val mutable next_footnote_number = 1

    method alloc_footnote n =
      let number = next_footnote_number in
      next_footnote_number <- number+1;
      footnotes <- footnotes @ [ number, n ];
      number

    method print_footnotes ch =
      if footnotes <> [] then begin
        output_string ch "<hr align=left noshade=noshade width=\"30%\">\n";
        output_string ch "<dl>\n";
        List.iter
          (fun (_,n) -> 
             n # footnote_to_html (self : #store_type :> store_type) ch)
          footnotes;
        output_string ch "</dl>\n";
      end

  end

The function escape_html

This function converts the characters <, >, &, and " to their HTML representations. For example,

 escape_html "<>" = "&lt;&gt;" 

Other characters are left unchanged.

let escape_html s =
  Str.global_substitute
    (Str.regexp "<\\|>\\|&\\|\"\\|@\\|:")
    (fun s ->
      match Str.matched_string s with
        "<" -> "&lt;"
      | ">" -> "&gt;"
      | "&" -> "&amp;"
      | "\"" -> "&quot;"
      | "@" -> "&#64;"
      | ":" -> "&#58;"
      | _ -> assert false)
    s

Note (of 2009): There is also the Ocamlnet function Netencoding.Html.encode one can use. It has a special XML mode.

The virtual class shared

This virtual class is the abstract superclass of the extension classes shown below. It defines the standard methods clone, node, and set_node, and declares the type of the virtual method to_html. This method recursively traverses the whole element tree, and prints the converted HTML code to the output channel passed as second argument. The first argument is the reference to the global store object which collects the footnotes.

class virtual shared =
  object (self)

    (* --- default_ext --- *)

    val mutable node = (None : shared node option)

    method clone = {< >} 
    method node =
      match node with
          None ->
            assert false
        | Some n -> n
    method set_node n =
      node <- Some n

    (* --- virtual --- *)

    method virtual to_html : store -> out_channel -> unit

  end

For an introduction into extension classes see Intro_extensions.

The class only_data

This class defines to_html such that the character data of the current node is converted to HTML. Note that self is an extension object (of type Pxp_document.extension), self # node is the node object (of type Pxp_document.node), and self # node # data returns the character data of the node (see Pxp_document.node.data).

class only_data =
  object (self)
    inherit shared

    method to_html store ch =
      output_string ch (escape_html (self # node # data))
  end

The class readme

This class converts elements of type readme to HTML. Such an element is (by definition) always the root element of the document. First, the HTML header is printed; the title attribute of the element determines the title of the HTML page. Some aspects of the HTML page can be configured by setting certain parameter entities, for example the background color, the text color, and link colors. After the header, the body tag, and the headline have been printed, the contents of the page are converted by invoking to_html on all children of the current node (which is the root node). Then, the footnotes are appended to this by telling the global store object to print the footnotes. Finally, the end tags of the HTML pages are printed.

class readme =
  object (self)
    inherit shared

    method to_html store ch =
      (* output header *)
      output_string 
        ch "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">";
      output_string
        ch "<!-- WARNING! This is a generated file, do not edit! -->\n";
      let title = 
        match self # node # attribute "title" with
            Value s -> s
          | _ -> assert false
      in
      let html_header =
        try replacement_text
            (self # node # dtd # par_entity "readme:html:header"
        with WF_error _ -> "" in
      let html_trailer =
        try replacement_text
            (self # node # dtd # par_entity "readme:html:trailer")
        with WF_error _ -> "" in
      let html_bgcolor =
        try replacement_text
            (self # node # dtd # par_entity "readme:html:bgcolor")
        with WF_error _ -> "white" in
      let html_textcolor =
        try replacement_text
            (self # node # dtd # par_entity "readme:html:textcolor")
        with WF_error _ -> "" in
      let html_alinkcolor =
        try replacement_text
            (self # node # dtd # par_entity "readme:html:alinkcolor")
        with WF_error _ -> "" in
      let html_vlinkcolor =
        try replacement_text
            (self # node # dtd # par_entity "readme:html:vlinkcolor")
        with WF_error _ -> "" in
      let html_linkcolor =
        try replacement_text 
            (self # node # dtd # par_entity "readme:html:linkcolor")
        with WF_error _ -> "" in
      let html_background =
        try replacement_text
            (self # node # dtd # par_entity "readme:html:background")
        with WF_error _ -> "" in

      output_string ch "<html><header><title>\n";
      output_string ch (escape_html title);
      output_string ch "</title></header>\n";
      output_string ch "<body ";
      List.iter
        (fun (name,value) ->
           if value <> "" then 
             output_string ch (name ^ "=\"" ^ escape_html value ^ "\" "))
        [ "bgcolor",    html_bgcolor;
          "text",       html_textcolor;
          "link",       html_linkcolor;
          "alink",      html_alinkcolor;
          "vlink",      html_vlinkcolor;
        ];
      output_string ch ">\n";
      output_string ch html_header;
      output_string ch "<h1>";
      output_string ch (escape_html title);
      output_string ch "</h1>\n";
      (* process main content: *)
      List.iter
        (fun n -> n # extension # to_html store ch)
        (self # node # sub_nodes);
      (* now process footnotes *)
      store # print_footnotes ch;
      (* trailer *)
      output_string ch html_trailer;
      output_string ch "</html>\n";
  end

This class is an example how to access the value of an attribute: The value is determined by invoking self # node # attribute "title" (see Pxp_document.node.attribute). As this attribute has been declared as CDATA and as being required, the value has always the form Value s where s is the string value of the attribute. Attribute values have type Pxp_core_types.S.att_value.

You can also see how entity contents can be accessed. A parameter entity object can be looked up by self # node # dtd # par_entity "name" (see Pxp_dtd.dtd.par_entity), and by invoking Pxp_dtd.Entity.replacement_text the value of the entity is returned after inner parameter and character entities have been processed. Note that you must use Pxp_dtd.dtd.gen_entity instead of par_entity to access general entities.

The classes section, sect1, sect2, and sect3

As the conversion process is very similar, the conversion classes of the three section levels are derived from the more general section class. The HTML code of the section levels only differs in the type of the headline, and because of this the classes describing the section levels can be computed by replacing the class argument the_tag of section by the HTML name of the headline tag.

class section the_tag =
  object (self)
    inherit shared

    val tag = the_tag

    method to_html store ch =
      let sub_nodes = self # node # sub_nodes in
      match sub_nodes with
          title_node :: rest ->
            output_string ch ("<" ^ tag ^ ">\n");
            title_node # extension # to_html store ch;
            output_string ch ("\n</" ^ tag ^ ">");
            List.iter
              (fun n -> n # extension # to_html store ch)
              rest
        | _ ->
            assert false
  end

class sect1 = section "h1"
class sect2 = section "h3"
class sect3 = section "h4"

Section elements are converted to HTML by printing a headline and then converting the contents of the element recursively. More precisely, the first sub-element is always a title element, and the other elements are the contents of the section. This structure is declared in the DTD, and it is guaranteed that the document matches the DTD. Because of this the title node can be separated from the rest without any checks.

Both the title node, and the body nodes are then converted to HTML by calling to_html on them.

The classes map_tag, p, em, ul, and li

Several element types are converted to HTML by simply mapping them to corresponding HTML element types. The class map_tag implements this, and the class argument the_target_tag determines the tag name to map to. The output consists of the start tag, the recursively converted inner elements, and the end tag.

class map_tag the_target_tag =
  object (self)
    inherit shared

    val target_tag = the_target_tag

    method to_html store ch =
      output_string ch ("<" ^ target_tag ^ ">\n");
      List.iter
        (fun n -> n # extension # to_html store ch)
        (self # node # sub_nodes);
      output_string ch ("\n</" ^ target_tag ^ ">");
  end

class p = map_tag "p"
class em = map_tag "b"
class ul = map_tag "ul"
class li = map_tag "li"

The class br

Element of type br are mapped to the same HTML type. Note that HTML forbids the end tag of br.

class br =
  object (self)
    inherit shared

    method to_html store ch =
      output_string ch "<br>\n";
      List.iter
        (fun n -> n # extension # to_html store ch)
        (self # node # sub_nodes);
  end

The class code

The code type is converted to a pre section (preformatted text). As the meaning of tabs is unspecified in HTML, tabs are expanded to spaces.

class code =
  object (self)
    inherit shared

    method to_html store ch =
      let data = self # node # data in
      (* convert tabs *)
      let l = String.length data in
      let rec preprocess i column =
        (* this is very ineffective but comprehensible: *)
        if i < l then
          match data.[i] with
              '\t' ->
                let n = 8 - (column mod 8) in
                String.make n ' ' ^ preprocess (i+1) (column + n)
            | '\n' ->
                "\n" ^ preprocess (i+1) 0
            | c ->
                String.make 1 c ^ preprocess (i+1) (column + 1)
        else
          ""
      in
      output_string ch "<p><pre>";
      output_string ch (escape_html (preprocess 0 0));
      output_string ch "</pre></p>";
  end

The class a

Hyperlinks, expressed by the a element type, are converted to the HTML a type. If the target of the hyperlink is given by href, the URL of this attribute can be used directly. Alternatively, the target can be given by readmeref in which case the ".html" suffix must be added to the file name.

Note that within a only #PCDATA is allowed, so the contents can be converted directly by applying escape_html to the character data contents.

class a =
  object (self)
    inherit shared

    method to_html store ch =
      output_string ch "<a ";
      let href =
        match self # node # attribute "href" with
            Value v -> escape_html v
          | Valuelist _ -> assert false
          | Implied_value ->
              begin match self # node # attribute "readmeref" with
                  Value v -> escape_html v ^ ".html"
                | Valuelist _ -> assert false
                | Implied_value ->
                    ""
              end
      in
      if href <> "" then
        output_string ch ("href=\""  ^ href ^ "\"");
      output_string ch ">";
      output_string ch (escape_html (self # node # data));
      output_string ch "</a>";
        
  end

The class footnote

The footnote class has two methods: to_html to convert the footnote reference to HTML, and footnote_to_html to convert the footnote text itself.

The footnote reference is converted to a local hyperlink; more precisely, to two anchor tags which are connected with each other. The text anchor points to the footnote anchor, and the footnote anchor points to the text anchor.

The footnote must be allocated in the store object. By allocating the footnote, you get the number of the footnote, and the text of the footnote is stored until the end of the HTML page is reached when the footnotes can be printed. The to_html method stores simply the object itself, such that the footnote_to_html method is invoked on the same object that encountered the footnote.

The to_html method only allocates the footnote, and prints the reference anchor, but it does not print nor convert the contents of the note. This is deferred until the footnotes actually get printed, i.e. the recursive call of to_html on the sub nodes is done by footnote_to_html

Note that this technique does not work if you make another footnote within a footnote; the second footnote gets allocated but not printed.

class footnote =
  object (self)
    inherit shared

    val mutable footnote_number = 0

    method to_html store ch =
      let number = 
        store # alloc_footnote (self : #shared :> footnote_printer) in
      let foot_anchor = 
        "footnote" ^ string_of_int number in
      let text_anchor =
        "textnote" ^ string_of_int number in
      footnote_number <- number;
      output_string ch ( "<a name=\"" ^ text_anchor ^ "\" href=\"#" ^ 
                         foot_anchor ^ "\">[" ^ string_of_int number ^ 
                         "]</a>" )

    method footnote_to_html store ch =
      (* prerequisite: we are in a definition list <dl>...</dl> *)
      let foot_anchor = 
        "footnote" ^ string_of_int footnote_number in
      let text_anchor =
        "textnote" ^ string_of_int footnote_number in
      output_string ch ("<dt><a name=\"" ^ foot_anchor ^ "\" href=\"#" ^ 
                        text_anchor ^ "\">[" ^ string_of_int footnote_number ^ 
                        "]</a></dt>\n<dd>");
      List.iter
        (fun n -> n # extension # to_html store ch)
        (self # node # sub_nodes);
      output_string ch ("\n</dd>")
 
  end

The specification of the document model

This code sets up the hash table that connects element types with the exemplars of the extension classes that convert the elements to HTML. See How to bind extension classes to element types for comments, and Pxp_document.make_spec_from_alist for the function definition.

let tag_map =
  make_spec_from_alist
    ~data_exemplar:(new data_impl (new only_data))
    ~default_element_exemplar:(new element_impl (new no_markup))
    ~element_alist:
      [ "readme", (new element_impl (new readme));
        "sect1",  (new element_impl (new sect1));
        "sect2",  (new element_impl (new sect2));
        "sect3",  (new element_impl (new sect3));
        "title",  (new element_impl (new no_markup));
        "p",      (new element_impl (new p));
        "br",     (new element_impl (new br));
        "code",   (new element_impl (new code));
        "em",     (new element_impl (new em));
        "ul",     (new element_impl (new ul));
        "li",     (new element_impl (new li));
        "footnote", (new element_impl (new footnote : #shared :> shared));
        "a",      (new element_impl (new a));
      ]
    ()

The readme converter to ASCII

This converter is quite similar to the HTML converter, and not presented here. The source code is part of the PXP source tarball, however.

The main program

open Pxp_types
open Pxp_document
open Pxp_tree_parser

let rec print_error e =
  prerr_endline(string_of_exn e)


let run f a =
  try f a with
      e -> print_error e


let convert_to_html filename =
  let document =
    parse_document_entity
      { default_config with encoding = `Enc_iso88591 }
      (from_file filename)
      To_html.tag_map
  in
  let root = document # root in
  let store = new To_html.store in
  root # extension # to_html store stdout


let convert_to_text filename =
  let document =
    parse_document_entity
      default_config
      (from_file filename)
      To_text.tag_map
  in
  let root = document # root in
  let store = new To_text.store in
  let box = new To_text.box 79 79 in
  root # extension # to_box store box;
  box # output 0 0 stdout


let main() =
  let want_html = ref false in
  let want_text = ref false in
  let filename = ref None in
  Arg.parse
      [ "-html"Arg.Set want_html, 
              "  convert file to html";
        "-text"Arg.Set want_text,
              "  convert file to text";
      ]
      (fun s -> 
         match !filename with
             None -> filename := Some s
           | Some _ ->
               raise (Arg.Bad "Multiple arguments not allowed."))
      "usage: readme [ -text | -html ] input.xml >output";
  let fn =
    match !filename with
        None -> 
          prerr_endline "readme: no input";
          exit 1
      | Some s -> s
  in
  match !want_html, !want_text with
      truefalse ->
        run convert_to_html fn
    | falsetrue ->
        run convert_to_text fn
    | _ ->
        prerr_endline ("readme: Please select exactly one output format")

let () =
  main()


This web site is published by Informatikbüro Gerd Stolpmann
Powered by Caml