******************************************************************************
Notes on the XML specification
******************************************************************************


==============================================================================
This document
==============================================================================

There are some points in the XML specification which are ambiguous. The 
following notes discuss these points, and describe how this parser behaves.

==============================================================================
Conditional sections and the token ]]>
==============================================================================

It is unclear what happens if an ignored section contains the token ]]> at 
places where it is normally allowed, i.e. within string literals and comments, 
e.g. 

<![IGNORE[ <!-- ]]> --> ]]>

On the one hand, the production rule of the XML grammar does not treat such 
tokens specially. Following the grammar, already the first ]]> ends the 
conditional section 

<![IGNORE[ <!-- ]]>

and the other tokens are included into the DTD.

On the other hand, we can read: "Like the internal and external DTD subsets, a 
conditional section may contain one or more complete declarations, comments, 
processing instructions, or nested conditional sections, intermingled with 
white space" (XML 1.0 spec, section 3.4). Complete declarations and comments 
may contain ]]>, so this is contradictory to the grammar.

The intention of conditional sections is to include or exclude the section 
depending on the current replacement text of a parameter entity. Almost always 
such sections are used as in 

<!ENTITY % want.a.feature.or.not "INCLUDE">   (or "IGNORE")
<![ %want.a.feature.or.not; [ ... ]]>

This means that if it is possible to include a section it must also be legal to 
ignore the same section. This is a strong indication that the token ]]> must 
not count as section terminator if it occurs in a string literal or comment.

This parser implements the latter.

==============================================================================
Conditional sections and the inclusion of parameter entities
==============================================================================

It is unclear what happens if an ignored section contains a reference to a 
parameter entity. In most cases, this is not problematic because nesting of 
parameter entities must respect declaration braces. The replacement text of 
parameter entities must either contain a whole number of declarations or only 
inner material of one declaration. Almost always it does not matter whether 
these references are resolved or not (the section is ignored).

But there is one case which is not explicitly specified: Is it allowed that the 
replacement text of an entity contains the end marker ]]> of an ignored 
conditional section? Example: 

<!ENTITY % end "]]>">
<![ IGNORE [ %end;

We do not find the statement in the XML spec that the ]]> must be contained in 
the same entity as the corresponding <![ (as for the tokens <! and > of 
declarations). So it is possible to conclude that ]]> may be in another entity.

Of course, there are many arguments not to allow such constructs: The resulting 
code is incomprehensive, and parsing takes longer (especially if the entities 
are external). I think the best argument against this kind of XML is that the 
XML spec is not detailed enough, as it contains no rules where entity 
references should be recognized and where not. For example: 

<!ENTITY % y "]]>">
<!ENTITY % x "<!ENTITY z '<![CDATA[some text%y;'>">
<![ IGNORE [ %x; ]]>

Which token ]]> counts? From a logical point of view, the ]]> in the third line 
ends the conditional section. As already pointed out, the XML spec permits the 
interpretation that ]]> is recognized even in string literals, and this may be 
also true if it is "imported" from a separate entity; and so the first ]]> 
denotes the end of the section.

As a practical solution, this parser does not expand parameter entities in 
ignored sections. Furthermore, it is also not allowed that the ending ]]> of 
ignored or included sections is contained in a different entity than the 
starting <![ token.

==============================================================================
Standalone documents and attribute normalization
==============================================================================

If a document is declared as stand-alone, a restriction on the effect of 
attribute normalization takes effect for attributes declared in external 
entities. Normally, the parser knows the type of the attribute from the ATTLIST 
declaration, and it can normalize attribute values depending on their types. 
For example, an NMTOKEN attribute can be written with leading or trailing 
spaces, but the parser returns always the nmtoken without such added spaces; in 
contrast to this, a CDATA attribute is not normalized in this way. For 
stand-alone document the type information is not available if the ATTLIST 
declaration is located in an external entity. Because of this, the XML spec 
demands that attribute values must be written in their normal form in this 
case, i.e. without additional spaces. 

This parser interprets this restriction as follows. Obviously, the substitution 
of character and entity references is not considered as a "change of the value" 
as a result of the normalization, because these operations will be performed 
identically if the ATTLIST declaration is not available. The same applies to 
the substitution of TABs, CRs, and LFs by space characters. Only the removal of 
spaces depending on the type of the attribute changes the value if the ATTLIST 
is not available. 

This means in detail: CDATA attributes never violate the stand-alone status. 
ID, IDREF, NMTOKEN, ENTITY, NOTATION and enumerator attributes must not be 
written with leading and/or trailing spaces. IDREF, ENTITIES, and NMTOKENS 
attributes must not be written with extra spaces at the beginning or at the end 
of the value, or between the tokens of the list. 

The whole check is dubious, because the attribute type expresses also a 
semantical constraint, not only a syntactical one. At least this parser 
distinguishes strictly between single-value and list types, and returns the 
attribute values differently; the first are represented as Value s (where s is 
a string), the latter are represented as Valuelist [s1; s2; ...; sN]. The 
internal representation of the value is dependent on the attribute type, too, 
such that even normalized values are processed differently depending on whether 
the attribute has list type or not. For this parser, it makes still a 
difference whether a value is normalized and processed as if it were CDATA, or 
whether the value is processed according to its declared type. 

The stand-alone check is included to be able to make a statement whether other, 
well-formedness parsers can process the document. Of course, these parsers 
always process attributes as CDATA, and the stand-alone check guarantees that 
these parsers will always see the normalized values. 

==============================================================================
Standalone documents and the restrictions on entity
references
==============================================================================

Stand-alone documents must not refer to entities which are declared in an 
external entity. This parser applies this rule only: to general and NDATA 
entities when they occur in the document body (i.e. not in the DTD); and to 
general and NDATA entities occuring in default attribute values declared in the 
internal subset of the DTD. 

Parameter entities are out of discussion for the stand-alone property. If there 
is a parameter entity reference in the internal subset which was declared in an 
external entity, it is not available in the same way as the external entity is 
not available that contains its declaration. Because of this "equivalence", 
parameter entity references are not checked on violations against the 
stand-alone declaration. It simply does not matter. - Illustration: 

Main document:  

<!ENTITY % ext SYSTEM "ext">
%ext;
%ent;

"ext" contains:  

<!ENTITY % ent "<!ELEMENT el (other*)>">



Here, the reference %ent; would be illegal if the standalone declaration is 
strictly interpreted. This parser handles the references %ent; and %ext; 
equivalently which means that %ent; is allowed, but the element type "el" is 
treated as externally declared. 

General entities can occur within the DTD, but they can only be contained in 
the default value of attributes, or in the definition of other general 
entities. The latter can be ignored, because the check will be repeated when 
the entities are expanded. Though, general entities occuring in default 
attribute values are actually checked at the moment when the default is used in 
an element instance. 

General entities occuring in the document body are always checked.

NDATA entities can occur in ENTITY attribute values; either in the element 
instance or in the default declaration. Both cases are checked.