formats: std/xml #78

Open
opened 2026-10-05 21:07:17 +00:00 by orion · 13 comments
Collaborator

Context

XML with its tree; S3 listings and errors need it (FIRST-PARTY gap 24), and the WebDAV driver (#67) needs it for multistatus replies. This is an epic: a whole format package with a reader, a writer, namespaces, entities, typed binding and a mapping DSL is more than one PR. Steps become issues just in time, one at a time.

Source

  • ROADMAP.md, format versions: XML 1.0 (fifth edition) with namespaces; a DTD's entities are read, an external entity is never fetched; XML 1.1 and validation against a DTD are not supported.
  • ENCODING.md section 9 and slice 8 (section 14): XmlNode as the tree, Xml.format { map<T> { ... } } as the mapping DSL (section 6, with its five-rule contract), no XmlEncode/XmlDecode.
  • std/toml is the layout model (format value with options, tree, parser, writer, vendored suite, docs page).

Design record point

ENCODING.md slice 8, sections 6 and 9; ROADMAP "Which versions a format supports". Touches the seed? No for the whole epic: the compiler does not import std/xml and no std name is looked up by string. A step that changes that carries the two-commit rule itself.

Steps

  • 1. Tree, reader and writer: XmlNode, Xml().parse/write, XmlError, XML 1.0 with namespaces, a DOCTYPE without internal subset, vendored namespace/well-formedness cases, docs page #429
  • 2. DTD entities: the internal subset's general entities are read (and the document's attribute defaults), with an expansion limit like std/yaml's alias limit (billion laughs is an error, not an out-of-memory panic), an external entity or DTD never fetched; hand-written cases from the Recommendation for what an internal subset can break (the W3C suite cannot be redistributed, see std/xml/tests/NOTICE.md) #552
  • 3a. Typed reading: Xml().decode<T>(text) with the default mapping (attribute or child element, repeated children as a list), naming, strict, an S3 ListBucketResult and Error decoded into types #570
  • 3b. Typed writing: Xml().encode(value) and Xml().tree(value) with the default mapping (a field is a child element), naming, XmlEncoder #605
  • 3c. Format<XmlError>: the trait's encodeAll and encoded cannot report a failure and Xml.encode can fail on data (PANICS.md forbids a panic for that), so the trait needs an additive failing twin or a lossy encodeAll; decide when it starts
  • 4. The mapping DSL: Xml.format { map<T> { element ...; attribute { _.field }; text { _.field } } } answering Result<Xml, XmlProblem>, validated when built (ENCODING.md section 6, rules 1 to 5); check first which of its parts need compiler support (quoted field references, typeName) and slice further then

Dependencies

  • Needs: std/encoding (built, slices 1 to 6). The mapping DSL (step 4) rests on quoted expressions and nameOf, which exist (std/expression); with Encode by value (ENCODING.md slice 7) is not a prerequisite.
  • Unblocks: #67 (WebDAV driver, step 1 for multistatus replies, step 3 for typed ones), #20 (s3: driver, steps 1 and 3 for listings and errors).
  • Independent of #79 (std/html, its own parsing algorithm and tree) and #80 (JSON Schema).

Moved from the planning lists of the docs; the design record stays the specification.

## Context XML with its tree; S3 listings and errors need it (FIRST-PARTY gap 24), and the WebDAV driver (#67) needs it for multistatus replies. This is an epic: a whole format package with a reader, a writer, namespaces, entities, typed binding and a mapping DSL is more than one PR. Steps become issues just in time, one at a time. ## Source - [ROADMAP.md, format versions](https://git.torb.dev/torbscript/language/src/commit/393ef2608e4a60b8ff76d8b6b408951b822d887d/docs/ROADMAP.md#L185): XML 1.0 (fifth edition) with namespaces; a DTD's entities are read, an external entity is never fetched; XML 1.1 and validation against a DTD are not supported. - [ENCODING.md section 9](https://git.torb.dev/torbscript/language/src/commit/393ef2608e4a60b8ff76d8b6b408951b822d887d/docs/design/ENCODING.md#L709) and slice 8 (section 14): `XmlNode` as the tree, `Xml.format { map<T> { ... } }` as the mapping DSL (section 6, with its five-rule contract), no `XmlEncode`/`XmlDecode`. - `std/toml` is the layout model (format value with options, tree, parser, writer, vendored suite, docs page). ## Design record point ENCODING.md slice 8, sections 6 and 9; ROADMAP "Which versions a format supports". Touches the seed? No for the whole epic: the compiler does not import `std/xml` and no std name is looked up by string. A step that changes that carries the two-commit rule itself. ## Steps - [x] 1. Tree, reader and writer: `XmlNode`, `Xml().parse`/`write`, `XmlError`, XML 1.0 with namespaces, a DOCTYPE without internal subset, vendored namespace/well-formedness cases, docs page #429 - [x] 2. DTD entities: the internal subset's general entities are read (and the document's attribute defaults), with an expansion limit like `std/yaml`'s alias limit (billion laughs is an error, not an out-of-memory panic), an external entity or DTD never fetched; hand-written cases from the Recommendation for what an internal subset can break (the W3C suite cannot be redistributed, see `std/xml/tests/NOTICE.md`) #552 - [x] 3a. Typed reading: `Xml().decode<T>(text)` with the default mapping (attribute or child element, repeated children as a list), `naming`, `strict`, an S3 `ListBucketResult` and `Error` decoded into types #570 - [ ] 3b. Typed writing: `Xml().encode(value)` and `Xml().tree(value)` with the default mapping (a field is a child element), `naming`, `XmlEncoder` #605 - [ ] 3c. `Format<XmlError>`: the trait's `encodeAll` and `encoded` cannot report a failure and `Xml.encode` can fail on data (PANICS.md forbids a panic for that), so the trait needs an additive failing twin or a lossy `encodeAll`; decide when it starts - [ ] 4. The mapping DSL: `Xml.format { map<T> { element ...; attribute { _.field }; text { _.field } } }` answering `Result<Xml, XmlProblem>`, validated when built (ENCODING.md section 6, rules 1 to 5); check first which of its parts need compiler support (quoted field references, `typeName`) and slice further then ## Dependencies - Needs: `std/encoding` (built, slices 1 to 6). The mapping DSL (step 4) rests on quoted expressions and `nameOf`, which exist (`std/expression`); `with Encode by value` (ENCODING.md slice 7) is not a prerequisite. - Unblocks: #67 (WebDAV driver, step 1 for multistatus replies, step 3 for typed ones), #20 (`s3:` driver, steps 1 and 3 for listings and errors). - Independent of #79 (`std/html`, its own parsing algorithm and tree) and #80 (JSON Schema). _Moved from the planning lists of the docs; the design record stays the specification._
Author
Collaborator

[loop:torbscript] intake: in scope (author orion, label loop), sources ROADMAP.md format versions and ENCODING.md section 9, not a duplicate, no collision with a design record. TorbScript package on Encode/Decode, needed by #67 and S3 listings; no seed impact expected. next: refine, waiting (ready buffer full: #4, #163, #164, #223 are ready) until 2026-10-10T04:00:00Z

[loop:torbscript] intake: in scope (author orion, label loop), sources ROADMAP.md format versions and ENCODING.md section 9, not a duplicate, no collision with a design record. TorbScript package on `Encode`/`Decode`, needed by #67 and S3 listings; no seed impact expected. next: refine, waiting (ready buffer full: #4, #163, #164, #223 are ready) until 2026-10-10T04:00:00Z
Author
Collaborator

[loop:torbscript] claim refine by session:loop-torbscript-20261023 until 2026-10-23T15:27Z

[loop:torbscript] claim refine by session:loop-torbscript-20261023 until 2026-10-23T15:27Z
Author
Collaborator

[loop:torbscript] decided by the loop: #78 is too big for one PR, so it is an epic and stays without ready. A reader with namespaces, entity handling with an expansion limit, a writer, typed Encode/Decode, and the mapping DSL of ENCODING.md section 6 are four reviewable pieces (std/toml alone is the size of step 1). The order follows the design record: the tree first (ENCODING.md step 6 of the ladder, what S3 listings need), then DTD entities, then the typed binding, then the DSL. No seed touch (the compiler does not import std/xml). Step 1 is filed as #429 (keyword search of open issues found only #20, #67 and this one; none duplicates it) and is ready; steps 2 to 4 are filed just in time. Found on the way: CONCEPT.md around line 2911 still says XML gets its own traits, against ENCODING.md and CONCEPT line 1768; #429 reconciles it.

[loop:torbscript] decided by the loop: #78 is too big for one PR, so it is an epic and stays without `ready`. A reader with namespaces, entity handling with an expansion limit, a writer, typed `Encode`/`Decode`, and the mapping DSL of ENCODING.md section 6 are four reviewable pieces (std/toml alone is the size of step 1). The order follows the design record: the tree first (ENCODING.md step 6 of the ladder, what S3 listings need), then DTD entities, then the typed binding, then the DSL. No seed touch (the compiler does not import `std/xml`). Step 1 is filed as #429 (keyword search of open issues found only #20, #67 and this one; none duplicates it) and is `ready`; steps 2 to 4 are filed just in time. Found on the way: CONCEPT.md around line 2911 still says XML gets its own traits, against ENCODING.md and CONCEPT line 1768; #429 reconciles it.
Author
Collaborator

[loop:torbscript] refine: released the claim of session:loop-torbscript-20261023. #78 is an epic (checklist of 4 steps in the body), step 1 filed as #429 with label ready. Nothing was written to the repo. Worker usage: about 85k tokens (sonnet).
next: waiting (epic waits on its steps)

[loop:torbscript] refine: released the claim of session:loop-torbscript-20261023. #78 is an epic (checklist of 4 steps in the body), step 1 filed as #429 with label `ready`. Nothing was written to the repo. Worker usage: about 85k tokens (sonnet). next: waiting (epic waits on its steps)
Author
Collaborator

[loop:torbscript] claim refine by session:loop-torbscript-20261024 until 2026-10-24T00:10Z

[loop:torbscript] claim refine by session:loop-torbscript-20261024 until 2026-10-24T00:10Z
Author
Collaborator

[loop:torbscript] release refine: epic already refined, its next step is its own issue; claim written in error. next: waiting (epic steps)

[loop:torbscript] release refine: epic already refined, its next step is its own issue; claim written in error. next: waiting (epic steps)
Author
Collaborator

[loop:torbscript] claim refine by session:loop-torbscript-20261028 until 2026-10-28T18:57:37Z

[loop:torbscript] claim refine by session:loop-torbscript-20261028 until 2026-10-28T18:57:37Z
Author
Collaborator

[loop:torbscript] decided by the loop: step 2 is one PR (two Conventional Commits: the subset reader with entity expansion and the limit, then the attribute-list defaults), filed as #552 with label ready. Choices recorded there: option entityLimit (characters added by entity references, default 1000000) modelled on aliasLimit; a reference to an external or unparsed entity is an XmlError and not skipped; the Doctype node does not keep the subset, so the tree stays a tree of the document and parse(write(tree)) == tree holds; the checklist wording "rest of the vendored W3C suite" is corrected to hand-written cases, because the suite cannot be redistributed (std/xml/tests/NOTICE.md from step 1). Keyword search of open issues (xml, entity, DTD) found no duplicate. Step 1 is ticked, step 2 carries #552; steps 3 and 4 follow just in time. No seed touch.
[loop:torbscript] refine: step 2 filed as #552 (ready), checklist updated, released the claim of session:loop-torbscript-20261028. Worker usage: about 95k tokens (sonnet), next: waiting (epic steps)

[loop:torbscript] decided by the loop: step 2 is one PR (two Conventional Commits: the subset reader with entity expansion and the limit, then the attribute-list defaults), filed as #552 with label `ready`. Choices recorded there: option `entityLimit` (characters added by entity references, default 1000000) modelled on `aliasLimit`; a reference to an external or unparsed entity is an `XmlError` and not skipped; the `Doctype` node does not keep the subset, so the tree stays a tree of the document and `parse(write(tree)) == tree` holds; the checklist wording "rest of the vendored W3C suite" is corrected to hand-written cases, because the suite cannot be redistributed (std/xml/tests/NOTICE.md from step 1). Keyword search of open issues (xml, entity, DTD) found no duplicate. Step 1 is ticked, step 2 carries #552; steps 3 and 4 follow just in time. No seed touch. [loop:torbscript] refine: step 2 filed as https://git.torb.dev/torbscript/language/issues/552 (ready), checklist updated, released the claim of session:loop-torbscript-20261028. Worker usage: about 95k tokens (sonnet), next: waiting (epic steps)
Author
Collaborator

[loop:torbscript] claim refine (next step after the landed one) by session:loop-torbscript-20261029 until 2026-10-29T08:35Z

[loop:torbscript] claim refine (next step after the landed one) by session:loop-torbscript-20261029 until 2026-10-29T08:35Z
Author
Collaborator

[loop:torbscript] decided by the loop: step 2 (#552) is closed and landed on main. Step 3 would be one PR too large (an encoder, a decoder, Format<XmlError> and the S3 cases), so it is split: 3a typed reading is filed as #570 (ready), because S3 listings and errors (#20) and WebDAV replies (#67) read XML first; 3b typed writing and Format<XmlError> follow just in time after 3a lands. Choices recorded in #570: names match on the local name (S3 uses a default namespace), a missing child element falls back to the attribute of that name, a single child reads as a one-item list through a new ValueDecoder option (commit 1), then the decoder (commit 2). Keyword search (xml, s3, binding, decode) found no duplicate. No seed touch. Checklist: step 2 ticked, step 3 split into 3a (#570) and 3b.

[loop:torbscript] decided by the loop: step 2 (#552) is closed and landed on `main`. Step 3 would be one PR too large (an encoder, a decoder, `Format<XmlError>` and the S3 cases), so it is split: 3a typed reading is filed as #570 (`ready`), because S3 listings and errors (#20) and WebDAV replies (#67) read XML first; 3b typed writing and `Format<XmlError>` follow just in time after 3a lands. Choices recorded in #570: names match on the local name (S3 uses a default namespace), a missing child element falls back to the attribute of that name, a single child reads as a one-item list through a new `ValueDecoder` option (commit 1), then the decoder (commit 2). Keyword search (xml, s3, binding, decode) found no duplicate. No seed touch. Checklist: step 2 ticked, step 3 split into 3a (#570) and 3b.
Author
Collaborator

[loop:torbscript] refine: step 3a filed as #570 (labels loop, ready), checklist of #78 updated (step 2 ticked, step 3 split), released the claim of session:loop-torbscript-20261029. Worker usage: about 90k tokens (sonnet). next: ready (#570)

[loop:torbscript] refine: step 3a filed as https://git.torb.dev/torbscript/language/issues/570 (labels `loop`, `ready`), checklist of #78 updated (step 2 ticked, step 3 split), released the claim of session:loop-torbscript-20261029. Worker usage: about 90k tokens (sonnet). next: ready (#570)
Author
Collaborator

[loop:torbscript] claim refine by session:loop-torbscript-20261029-0858 until 2026-10-29T20:45:00Z

[loop:torbscript] claim refine by session:loop-torbscript-20261029-0858 until 2026-10-29T20:45:00Z
Author
Collaborator

[loop:torbscript] decided by the loop: step 3a (#570) is closed and landed on main (ebb4de3e singleAsList option, cd8da8b5 Xml.decode; main CI success; std/xml/src/xml.trb has decode, naming, strict). The seed is not involved: the compiler does not import std/xml. Step 3b is filed as #605 (loop, ready): Xml.encode and Xml.tree with the default mapping and a new XmlEncoder, modelled on TomlEncoder; the root element is the record's type name, a field a child element, None left out, a sequence repeated elements; two commits (encoder, then encode with docs). Format<XmlError> is split off as its own step 3c: the Format trait (STREAMS.md section 10) has encodeAll/encoded that never fail, Xml.encode can fail on data (a NUL in a string, a map key), and PANICS.md forbids a panic for a failure that depends on the data, so the trait needs a decision first (additive failing twin with a default, or a lossy encodeAll only); that is filed just in time after #605 lands. Keyword search of all issues (xml, encode, Format) found no duplicate. Checklist: 3a ticked with #570, 3b carries #605, 3c added.
[loop:torbscript] release refine: step 3b filed as #605 (labels loop, ready), checklist of #78 updated, released the claim of session:loop-torbscript-20261029-0858. Nothing was written to the repo. Worker usage: about 100k tokens (sonnet). next: ready (#605)

[loop:torbscript] decided by the loop: step 3a (#570) is closed and landed on `main` (`ebb4de3e` `singleAsList` option, `cd8da8b5` `Xml.decode`; `main` CI success; `std/xml/src/xml.trb` has `decode`, `naming`, `strict`). The seed is not involved: the compiler does not import `std/xml`. Step 3b is filed as #605 (`loop`, `ready`): `Xml.encode` and `Xml.tree` with the default mapping and a new `XmlEncoder`, modelled on `TomlEncoder`; the root element is the record's type name, a field a child element, `None` left out, a sequence repeated elements; two commits (encoder, then encode with docs). `Format<XmlError>` is split off as its own step 3c: the `Format` trait (STREAMS.md section 10) has `encodeAll`/`encoded` that never fail, `Xml.encode` can fail on data (a NUL in a string, a map key), and PANICS.md forbids a panic for a failure that depends on the data, so the trait needs a decision first (additive failing twin with a default, or a lossy `encodeAll` only); that is filed just in time after #605 lands. Keyword search of all issues (xml, encode, Format) found no duplicate. Checklist: 3a ticked with #570, 3b carries #605, 3c added. [loop:torbscript] release refine: step 3b filed as https://git.torb.dev/torbscript/language/issues/605 (labels `loop`, `ready`), checklist of #78 updated, released the claim of session:loop-torbscript-20261029-0858. Nothing was written to the repo. Worker usage: about 100k tokens (sonnet). next: ready (#605)
Sign in to join this conversation.
No labels
loop
ready
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
torbscript/language#78
No description provided.