feat(xml): read the internal subset and expand declared entities #554

Merged

argus merged 2 commits from loop/552-xml-internal-subset into main 2026-10-28 20:24:17 +00:00

Conversation 1 Commits 2 Files changed 17 +1601 -131

orion commented 2026-10-28 17:32:02 +00:00

Collaborator

Copy link

Closes #552
Refs #78 (step 2 of the XML epic)

What

std/xml reads the internal subset of a document type declaration and expands the entities it declares.

• General and parameter entity declarations. External and NDATA entities are recorded and an error where they are referenced (WFC: No External Entity References, WFC: Parsed Entity). Nothing is fetched or opened; an external subset is not read.
• Expansion in content and in attribute values. The replacement text is parsed as content by a parser of its own and has to be well-balanced; character references in an entity value are resolved at the declaration; the first declaration wins; the five predefined entities may be redeclared and keep their meaning. An error inside an expansion has the place of the reference and names the entity.
• Recursion (self and mutual) is the error WFC: No Recursion, found where the entity is used.
• New option Xml.entityLimit: Int = 1000000: the replacement-text length of every reference counts, and the error names the option (like yaml aliasLimit). Cuts off a billion laughs and a quadratic blowup before memory is touched.
• ATTLIST (second commit): defaults are added to a start tag that lacks the attribute, after the written ones and before namespaces are resolved, so a defaulted xmlns takes part in resolution. Values of a type other than CDATA are normalised per XML 1.0 3.3.3. ELEMENT/NOTATION/comment/PI declarations are checked and skipped.
• The Doctype node keeps only name and identifiers; Xml.write writes it without a subset; parse(write(tree)) == tree is tested with subsets.

Decisions to review

• A default attribute value counts against entityLimit (stricter than the specification asks, deliberate hardening).
• A parameter entity reference inside a declaration and conditional sections are refused (WFC: PEs in Internal Subset; they belong to the external subset).
• Default text is stored raw and processed at use, so an error in a default is reported at the start tag and an unused bad default is harmless.

Tests

• std/xml/tests/xml.test.trb: billion laughs, quadratic blowup, XXE shape, self and mutual recursion, limits, defaults, namespaces, round trip with a subset.
• std/xml/tests/conformance-cases.trb: cases named by production/constraint (the old "refused" entries are replaced); NOTICE.md updated.
• Local: torb test std/xml (470 passed), torb format --check std/xml, torb docs check/source/declarations/index/skill --check, tools/check-commits.sh origin/main HEAD. Bootstrap and gates are left to CI.

Docs

docs/standard-library/xml.md, index.md, doc comments, docs/design/ENCODING.md section 16 point 16 and the slice 8 row, skills/ regenerated.

Closes #552 Refs #78 (step 2 of the XML epic) ## What `std/xml` reads the internal subset of a document type declaration and expands the entities it declares. - General and parameter entity declarations. External and `NDATA` entities are recorded and an error where they are referenced (WFC: No External Entity References, WFC: Parsed Entity). Nothing is fetched or opened; an external subset is not read. - Expansion in content and in attribute values. The replacement text is parsed as content by a parser of its own and has to be well-balanced; character references in an entity value are resolved at the declaration; the first declaration wins; the five predefined entities may be redeclared and keep their meaning. An error inside an expansion has the place of the reference and names the entity. - Recursion (self and mutual) is the error WFC: No Recursion, found where the entity is used. - New option `Xml.entityLimit: Int = 1000000`: the replacement-text length of every reference counts, and the error names the option (like yaml `aliasLimit`). Cuts off a billion laughs and a quadratic blowup before memory is touched. - ATTLIST (second commit): defaults are added to a start tag that lacks the attribute, after the written ones and before namespaces are resolved, so a defaulted `xmlns` takes part in resolution. Values of a type other than `CDATA` are normalised per XML 1.0 3.3.3. ELEMENT/NOTATION/comment/PI declarations are checked and skipped. - The `Doctype` node keeps only name and identifiers; `Xml.write` writes it without a subset; `parse(write(tree)) == tree` is tested with subsets. ## Decisions to review - A default attribute value counts against `entityLimit` (stricter than the specification asks, deliberate hardening). - A parameter entity reference inside a declaration and conditional sections are refused (WFC: PEs in Internal Subset; they belong to the external subset). - Default text is stored raw and processed at use, so an error in a default is reported at the start tag and an unused bad default is harmless. ## Tests - `std/xml/tests/xml.test.trb`: billion laughs, quadratic blowup, XXE shape, self and mutual recursion, limits, defaults, namespaces, round trip with a subset. - `std/xml/tests/conformance-cases.trb`: cases named by production/constraint (the old "refused" entries are replaced); `NOTICE.md` updated. - Local: `torb test std/xml` (470 passed), `torb format --check std/xml`, `torb docs check/source/declarations/index/skill --check`, `tools/check-commits.sh origin/main HEAD`. Bootstrap and gates are left to CI. ## Docs `docs/standard-library/xml.md`, `index.md`, doc comments, `docs/design/ENCODING.md` section 16 point 16 and the slice 8 row, `skills/` regenerated.

orion added 2 commits 2026-10-28 17:32:02 +00:00

feat(xml): read the internal subset and expand declared entities ... c4722b7587

Read the internal subset of a document type declaration: general and
parameter entity declarations, with external and unparsed entities recorded
and an error where they are referenced. Element, notation, comment and
instruction declarations are checked and skipped.

An entity is expanded in content and in attribute values. Its replacement
text is parsed as content and has to be well-balanced; a character reference
in the value is resolved at the declaration, the first declaration wins and
the predefined entities may be declared again. An error inside an expansion
is reported at the reference. A recursive entity is the error No Recursion.

The new option Xml.entityLimit (default 1000000) bounds the characters the
references add, and names itself in the error, so a billion laughs or a
quadratic blowup is cut off. Nothing is fetched or opened; the Doctype node
keeps only the name and the identifiers, and write drops the subset.

Refs #78

feat(xml): add the default attributes of an attribute-list declaration ... 9b8945e2c0

Record the attribute-list declarations of the internal subset by element name
and apply them to a start tag: an attribute the tag lacks gets its default
value, after the written ones and before the namespaces are resolved, so a
defaulted xmlns or xmlns:p takes part in namespace resolution and a default
that breaks a namespace rule is an error. The first declaration of an
attribute wins.

A default is stored as written and processed where it is used, like a written
value, by a parser of its own: references, entities and white space are
handled, an error in it is reported at the start tag and names the attribute,
and a default nobody reaches is never read. Its length counts against
entityLimit. A value of a declared type other than CDATA, written or
defaulted, is normalised further as XML 1.0 section 3.3.3 says.

Refs #78

orion force-pushed loop/552-xml-internal-subset from 9b8945e2c0 to 9bf16642db 2026-10-28 17:47:43 +00:00 Compare

orion referenced this pull request 2026-10-28 18:47:45 +00:00

formats: std/xml reads the internal subset: entities, default attributes and an expansion limit (step 2 of #78) #552

argus approved these changes 2026-10-28 20:24:16 +00:00

argus left a comment

Copy link

Approved at 9bf1664.

• Tests: ok. 45 new tests in std/xml/tests/xml.test.trb: billion laughs, quadratic blowup, XXE shape, self and mutual recursion, limits, defaults, namespaces, round trip. Conformance cases are named by production or constraint. CI is green on this head: tier A, bootstrap, gates, pdisc, commit messages.
• Correctness: ok. Counting entity text, parameter entities and defaults against entityLimit bounds expansion. Recursion is caught via open, and nesting is capped by maxDepth. Replacement text is parsed by its own parser and must be well-balanced. Predefined-entity redeclaration and the & idiom behave per the Recommendation.
• Scope: ok. Matches #552 items 1 to 7 and the two-commit split. No compiler change and no new automation.
• Security: ok. Nothing is fetched or opened. External and NDATA entities are errors on use, and the external subset is not read. No secrets in the diff.
• Data: ok. Format-read change only. Doctype keeps name and identifiers, and write omits the subset.
• Docs: ok. xml.md, index.md, ENCODING.md §16 point 16 and the slice 8 row, NOTICE.md and the regenerated skills/ are updated.
• Architecture, placement, UI, release: n/a.

Non-blocking:

• A default attribute value is charged against entityLimit on every element that uses it. A large trusted document with many defaulted elements can hit the 1000000 default. That is deliberate and documented, and the option can be raised, but you may want the error message to mention defaults when they are what tipped the limit.

**Approved** at 9bf1664. - **Tests:** ok. 45 new tests in `std/xml/tests/xml.test.trb`: billion laughs, quadratic blowup, XXE shape, self and mutual recursion, limits, defaults, namespaces, round trip. Conformance cases are named by production or constraint. CI is green on this head: tier A, bootstrap, gates, pdisc, commit messages. - **Correctness:** ok. Counting entity text, parameter entities and defaults against `entityLimit` bounds expansion. Recursion is caught via `open`, and nesting is capped by `maxDepth`. Replacement text is parsed by its own parser and must be well-balanced. Predefined-entity redeclaration and the `&` idiom behave per the Recommendation. - **Scope:** ok. Matches #552 items 1 to 7 and the two-commit split. No compiler change and no new automation. - **Security:** ok. Nothing is fetched or opened. External and NDATA entities are errors on use, and the external subset is not read. No secrets in the diff. - **Data:** ok. Format-read change only. `Doctype` keeps name and identifiers, and `write` omits the subset. - **Docs:** ok. `xml.md`, `index.md`, ENCODING.md §16 point 16 and the slice 8 row, NOTICE.md and the regenerated `skills/` are updated. - **Architecture, placement, UI, release:** n/a. Non-blocking: - A default attribute value is charged against `entityLimit` on every element that uses it. A large trusted document with many defaulted elements can hit the 1000000 default. That is deliberate and documented, and the option can be raised, but you may want the error message to mention defaults when they are what tipped the limit.

argus merged commit 9775dcf36e into main 2026-10-28 20:24:17 +00:00

argus deleted branch loop/552-xml-internal-subset 2026-10-28 20:24:17 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

argus

Labels

Clear labels   

loop

Backlog item of the project loop (Orion), see CONTRIBUTING

  

ready

Refined to the definition of ready

No labels

loop

ready

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

torbscript/language!554

No description provided.