formats: `std/xml` reads the internal subset: entities, default attributes and an expansion limit (step 2 of #78) #552

New issue

Closed

opened 2026-10-28 16:59:02 +00:00 by orion · 2 comments

orion commented

2026-10-28 16:59:02 +00:00

Collaborator

Context

Step 2 of the epic #78 (std/xml). Step 1 (#429) landed the tree, the reader and the writer, and the reader refuses a document type declaration with an internal subset, because reading one means declaring entities. This step reads the internal subset, so that a document that declares its own entities and default attributes is read correctly, with an expansion limit so that a hostile document is an XmlError and not an out-of-memory panic, and with no external entity or DTD ever fetched or opened.

Source

ROADMAP.md, format versions: "XML 1.0 (fifth edition) with namespaces; a DTD's entities are read, an external entity is never fetched"; validation against a DTD is not supported.
ENCODING.md section 16, point 16: the refusal of an internal subset that this step replaces; point 15 (entity references are resolved and not kept) stays.
std/xml parser, doctype() and scanner, reference(): the two places that refuse today.
std/yaml's aliasLimit (yaml.trb, docs) is the model for the limit.
XML 1.0 (fifth edition) sections 2.8, 3.3, 4.1 to 4.7; Namespaces in XML 1.0.

Design record point

ENCODING.md section 16 point 16 (this issue changes it from "refused" to "read") and ROADMAP "Which versions a format supports". Touches the seed? No: the compiler does not import std/xml and no std name is looked up by string.

Scope

In:

The internal subset of the document type declaration is read: general entity declarations (internal, with literal replacement text; external SYSTEM/PUBLIC and unparsed NDATA ones are declared and recorded), parameter entity declarations and their references between declarations, attribute-list declarations, and <!ELEMENT>/<!NOTATION> declarations, comments and processing instructions, which are checked for well-formedness and otherwise skipped (no validation).
A reference to a declared internal general entity in content and in attribute values expands to its replacement text (content: parsed as content, so it may hold elements and must be well-balanced; attribute values: normalised as XML 1.0 section 3.3.3). Character references in an entity value are resolved when the entity is declared, general entity references are bypassed until use. The first declaration of an entity wins; the five predefined entities may be redeclared as the Recommendation says.
The default attributes of <!ATTLIST> are added to an element that does not carry them (#FIXED and plain defaults; #REQUIRED/#IMPLIED add nothing), and a declared tokenised type (ID, NMTOKEN, enumerations, ...) normalises the value as section 3.3.3 says. The first declaration of an attribute wins. A defaulted xmlns/xmlns:p attribute takes part in namespace resolution.
A limit option entityLimit: Int on Xml (decision, default 1000000): the total number of characters that references to declared entities add to one document. Exceeding it is an XmlError that names the option, like aliasLimit. Recursion (<!ENTITY a "&a;">, directly or in a cycle) is the well-formedness error "No Recursion", never a loop.
Nothing is fetched or opened: an external subset (SYSTEM/PUBLIC identifier on the doctype) is kept as the Doctype node's identifiers and not read; a reference to an external or unparsed entity in content or an attribute value is an XmlError that says the entity is not read (decision: error, not skip, since skipping would read the rest of the document wrongly). A reference to an entity that is not declared is an error as before, and the message says when the document has an external subset that this reader does not read.
The tree stays a tree of the document: the Doctype node keeps its name and identifiers and does not keep the subset (decision); entity references are resolved and not kept (ENCODING.md point 15). Xml.write therefore writes the doctype without a subset, and parse(write(tree)) == tree still holds for every tree parse made, defaults included, which are explicit attributes by then.
An XmlError for a problem inside an expansion reports the position of the reference in the document.

Out: validation of any kind (content models, ID uniqueness, IDREF resolution); reading an external subset or external entity; XML 1.1; the typed binding (step 3) and the mapping DSL (step 4).

Acceptance criteria

A document with an internal subset that declares general entities and default attributes parses; &name; in text and in attribute values reads as the replacement text, an entity with markup in its value yields the elements, and a default attribute shows up in attributes() and in attribute(local).
A billion-laughs document (ten levels of ten) and a quadratic-blowup document (a large entity referenced many times) are an XmlError naming entityLimit and finish quickly; raising entityLimit lets a legitimate larger document through. A self-referencing and a mutually recursive entity are an XmlError.
An external-entity document of the XXE shape (<!ENTITY x SYSTEM "file:///some/path"> and &x;) is an XmlError and nothing is opened: a test shows no file read happens (no file system or network access in the package, as in step 1).
Well-formedness constraints of the Recommendation that an internal subset can break are cases in std/xml/tests/conformance-cases.trb, each named by its production or constraint like the existing ones: DeclSep and markupdecl placement, PE Between Declarations, PEs in Internal Subset, Entity Declared, Parsed Entity, No Recursion, No External Entity References, No < in Attribute Values, Legal Character in a replacement text, Well-Formed Parsed Entity (unbalanced replacement text), Unique Att Spec with defaults, plus the namespace constraints on a defaulted xmlns. The W3C suite itself cannot be redistributed (see std/xml/tests/NOTICE.md); the cases are written from the Recommendation, and NOTICE.md is updated where it says cases that need a DTD are not covered.
parse(write(tree)) == tree holds for a parsed document with an internal subset.
Docs: docs/standard-library/xml.md (the "What the reader does" paragraph, the Xml declaration block with entityLimit, the sentence "refuses an internal subset" in the summary), the doc comments in xml.trb/parser.trb, ENCODING.md section 16 point 16 and the slice 8 status row, ROADMAP only if its XML line needs the wording; skills/ regenerated.
torb format --check on touched files, tier A/B and gates green in forge CI.

Size

One PR, two Conventional Commits, a std package change and no compiler change:

feat(xml): read the internal subset and expand declared entities (subset reader, entity table, expansion in content and attribute values, entityLimit, recursion and external-entity errors, tests);
feat(xml): add the default attributes of an attribute-list declaration (ATTLIST, defaults, tokenised-type normalisation, namespace interplay, tests, docs).

Estimate: 4 to 6 h of work, about 600 lines of parser and tests (the step 1 reader is 436 lines).

Dependencies

Needs: #429 (landed).
Part of #78. Unblocks step 3 of #78 (typed binding on Encode/Decode); the WebDAV (#67) and s3: (#20) drivers need only step 1 for their replies and are not blocked by this one.

Part of #78.

## Context Step 2 of the epic #78 (`std/xml`). Step 1 (#429) landed the tree, the reader and the writer, and the reader refuses a document type declaration with an internal subset, because reading one means declaring entities. This step reads the internal subset, so that a document that declares its own entities and default attributes is read correctly, with an expansion limit so that a hostile document is an `XmlError` and not an out-of-memory panic, and with no external entity or DTD ever fetched or opened. ## Source - [ROADMAP.md, format versions](https://git.torb.dev/torbscript/language/src/commit/5b87b231c24fc7a184f774df4b41648cad0564f5/docs/ROADMAP.md#L223): "XML 1.0 (fifth edition) with namespaces; a DTD's entities are read, an external entity is never fetched"; validation against a DTD is not supported. - [ENCODING.md section 16, point 16](https://git.torb.dev/torbscript/language/src/commit/5b87b231c24fc7a184f774df4b41648cad0564f5/docs/design/ENCODING.md#L1068): the refusal of an internal subset that this step replaces; point 15 (entity references are resolved and not kept) stays. - [std/xml parser, `doctype()`](https://git.torb.dev/torbscript/language/src/commit/5b87b231c24fc7a184f774df4b41648cad0564f5/std/xml/src/parser.trb#L167) and [scanner, `reference()`](https://git.torb.dev/torbscript/language/src/commit/5b87b231c24fc7a184f774df4b41648cad0564f5/std/xml/src/scanner.trb#L144): the two places that refuse today. - `std/yaml`'s `aliasLimit` ([yaml.trb](https://git.torb.dev/torbscript/language/src/commit/5b87b231c24fc7a184f774df4b41648cad0564f5/std/yaml/src/yaml.trb#L67), [docs](https://git.torb.dev/torbscript/language/src/commit/5b87b231c24fc7a184f774df4b41648cad0564f5/docs/standard-library/yaml.md#L98)) is the model for the limit. - XML 1.0 (fifth edition) sections 2.8, 3.3, 4.1 to 4.7; Namespaces in XML 1.0. ## Design record point ENCODING.md section 16 point 16 (this issue changes it from "refused" to "read") and ROADMAP "Which versions a format supports". Touches the seed? No: the compiler does not import `std/xml` and no std name is looked up by string. ## Scope In: 1. The internal subset of the document type declaration is read: general entity declarations (internal, with literal replacement text; external `SYSTEM`/`PUBLIC` and unparsed `NDATA` ones are declared and recorded), parameter entity declarations and their references between declarations, attribute-list declarations, and `<!ELEMENT>`/`<!NOTATION>` declarations, comments and processing instructions, which are checked for well-formedness and otherwise skipped (no validation). 2. A reference to a declared internal general entity in content and in attribute values expands to its replacement text (content: parsed as content, so it may hold elements and must be well-balanced; attribute values: normalised as XML 1.0 section 3.3.3). Character references in an entity value are resolved when the entity is declared, general entity references are bypassed until use. The first declaration of an entity wins; the five predefined entities may be redeclared as the Recommendation says. 3. The default attributes of `<!ATTLIST>` are added to an element that does not carry them (`#FIXED` and plain defaults; `#REQUIRED`/`#IMPLIED` add nothing), and a declared tokenised type (`ID`, `NMTOKEN`, enumerations, ...) normalises the value as section 3.3.3 says. The first declaration of an attribute wins. A defaulted `xmlns`/`xmlns:p` attribute takes part in namespace resolution. 4. A limit option `entityLimit: Int` on `Xml` (decision, default 1000000): the total number of characters that references to declared entities add to one document. Exceeding it is an `XmlError` that names the option, like `aliasLimit`. Recursion (`<!ENTITY a "&a;">`, directly or in a cycle) is the well-formedness error "No Recursion", never a loop. 5. Nothing is fetched or opened: an external subset (`SYSTEM`/`PUBLIC` identifier on the doctype) is kept as the `Doctype` node's identifiers and not read; a reference to an external or unparsed entity in content or an attribute value is an `XmlError` that says the entity is not read (decision: error, not skip, since skipping would read the rest of the document wrongly). A reference to an entity that is not declared is an error as before, and the message says when the document has an external subset that this reader does not read. 6. The tree stays a tree of the document: the `Doctype` node keeps its name and identifiers and does not keep the subset (decision); entity references are resolved and not kept (ENCODING.md point 15). `Xml.write` therefore writes the doctype without a subset, and `parse(write(tree)) == tree` still holds for every tree `parse` made, defaults included, which are explicit attributes by then. 7. An `XmlError` for a problem inside an expansion reports the position of the reference in the document. Out: validation of any kind (content models, `ID` uniqueness, `IDREF` resolution); reading an external subset or external entity; XML 1.1; the typed binding (step 3) and the mapping DSL (step 4). ## Acceptance criteria - A document with an internal subset that declares general entities and default attributes parses; `&name;` in text and in attribute values reads as the replacement text, an entity with markup in its value yields the elements, and a default attribute shows up in `attributes()` and in `attribute(local)`. - A billion-laughs document (ten levels of ten) and a quadratic-blowup document (a large entity referenced many times) are an `XmlError` naming `entityLimit` and finish quickly; raising `entityLimit` lets a legitimate larger document through. A self-referencing and a mutually recursive entity are an `XmlError`. - An external-entity document of the XXE shape (`<!ENTITY x SYSTEM "file:///some/path">` and `&x;`) is an `XmlError` and nothing is opened: a test shows no file read happens (no file system or network access in the package, as in step 1). - Well-formedness constraints of the Recommendation that an internal subset can break are cases in `std/xml/tests/conformance-cases.trb`, each named by its production or constraint like the existing ones: DeclSep and markupdecl placement, `PE Between Declarations`, `PEs in Internal Subset`, `Entity Declared`, `Parsed Entity`, `No Recursion`, `No External Entity References`, `No < in Attribute Values`, `Legal Character` in a replacement text, `Well-Formed Parsed Entity` (unbalanced replacement text), `Unique Att Spec` with defaults, plus the namespace constraints on a defaulted `xmlns`. The W3C suite itself cannot be redistributed (see `std/xml/tests/NOTICE.md`); the cases are written from the Recommendation, and NOTICE.md is updated where it says cases that need a DTD are not covered. - `parse(write(tree)) == tree` holds for a parsed document with an internal subset. - Docs: `docs/standard-library/xml.md` (the "What the reader does" paragraph, the `Xml` declaration block with `entityLimit`, the sentence "refuses an internal subset" in the summary), the doc comments in `xml.trb`/`parser.trb`, ENCODING.md section 16 point 16 and the slice 8 status row, ROADMAP only if its XML line needs the wording; `skills/` regenerated. - `torb format --check` on touched files, tier A/B and gates green in forge CI. ## Size One PR, two Conventional Commits, a std package change and no compiler change: 1. `feat(xml): read the internal subset and expand declared entities` (subset reader, entity table, expansion in content and attribute values, `entityLimit`, recursion and external-entity errors, tests); 2. `feat(xml): add the default attributes of an attribute-list declaration` (ATTLIST, defaults, tokenised-type normalisation, namespace interplay, tests, docs). Estimate: 4 to 6 h of work, about 600 lines of parser and tests (the step 1 reader is 436 lines). ## Dependencies - Needs: #429 (landed). - Part of #78. Unblocks step 3 of #78 (typed binding on `Encode`/`Decode`); the WebDAV (#67) and `s3:` (#20) drivers need only step 1 for their replies and are not blocked by this one. Part of #78.

orion added the

loop

ready

labels

2026-10-28 16:59:02 +00:00

orion referenced this issue

2026-10-28 16:59:10 +00:00

formats: std/xml #78

orion referenced this issue

2026-10-28 16:59:10 +00:00

formats: std/xml #78

orion commented

2026-10-28 17:06:32 +00:00

Author

Collaborator

[loop:torbscript] claim implement by session:loop-torbscript-20261028 until 2026-10-28T23:06:32Z

orion referenced this issue from a pull request that will close it,

2026-10-28 17:32:02 +00:00

feat(xml): read the internal subset and expand declared entities #554

orion commented

2026-10-28 18:47:45 +00:00

Author

Collaborator

[loop:torbscript] implement: std/xml reads the internal subset of the DOCTYPE (entities, ATTLIST defaults, entityLimit, tests, docs, skills); PR #554 is open, bootstrap is green and the rest of CI is queued on the runners, next: waiting (CI queue)

[loop:torbscript] implement: std/xml reads the internal subset of the DOCTYPE (entities, ATTLIST defaults, entityLimit, tests, docs, skills); PR https://git.torb.dev/torbscript/language/pulls/554 is open, bootstrap is green and the rest of CI is queued on the runners, next: waiting (CI queue)