dns: the rest of UTS #46 - mapping table, NFC, CONTEXTJ/CONTEXTO and Bidi (DNS.md slice 9) #365

New issue

Open

opened 2026-10-13 22:53:27 +00:00 by orion · 1 comment

orion commented

2026-10-13 22:53:27 +00:00

Collaborator

Context

DNS.md slice 9: "the rest of UTS #46: the mapping table, NFC, the combining-mark, CONTEXTJ, CONTEXTO and Bidi rules". Today DomainName.tryFrom lower-cases with Char.toLowerCase and refuses what it cannot check; the table in section 3 ("What is not covered, exactly") says each row is "closed when [the table] does". This issue closes all of them. Last of the chain of #36.

Depends on

#363 (String.toNfc) merged, and
#364 merged and in a published seed.

Not ready before both.

Decisions (refinement)

The processing stays what DNS.md section 3 says: UTS #46 lookup processing, non-transitional, UseSTD3ASCIIRules=true, CheckHyphens=false; now also CheckBidi=true and CheckJoiners=true, VerifyDnsLength=true. The section is rewritten to the as-built state; the "What is not covered" table is deleted and its rows move into the as-built text.
The step order of UTS #46 section 4: map (table, deviation characters ß ς kept, ZWJ/ZWNJ kept for CONTEXTJ), normalize to NFC, break at the dots, convert/validate each label (status check against the table, no combining mark first, hyphen rules as now, CONTEXTJ, CONTEXTO, Bidi rule per RFC 5893 on a domain that has a right-to-left label), Punycode, lengths.
Behaviour changes are the documented plan, not a breaking change of a promise: names with a disallowed code point that were encoded before are now refused with DnsError.InvalidName and a reason that names the code point (U+XXXX); upper-case non-Latin names are mapped (ПРИМЕР.рф becomes xn--e1afmkfd.xn--p1ai). DomainName.ofPresentation and ofLabels are untouched.
A reason string per refusal reuses the shape of refusal in std/dns/src/name.trb; no new DnsError case (the DnsError newtype question of RELEASE.md section 2 stays separate).
The old hand-made mappedPoint/isIgnored/refusal ranges are deleted, not kept as a fallback.

Acceptance criteria

DomainName.tryFrom implements the steps above on top of String.toNfc() and the private natives of the IDNA issue; std/dns/src/name.trb loses the hand-written ranges.
IdnaTestV2.txt of the pinned version passes in full for the to-ASCII column with the settings above (cases the file marks as errors are refused, others give the expected A-labels): a std test runs a committed excerpt of at least 400 lines chosen to cover every status, every error class the file uses and each CONTEXTO rule; the whole file runs in CI after tools/fetch-unicode.sh.
Tests in std/dns/tests/names.test.trb: the Cyrillic and Greek upper-case names, bu + U+0308 equal to bü, a leading combining mark refused, ß/ς kept, a Persian name that needs ZWNJ accepted and a stray ZWNJ refused, the middle dot rule (l·l accepted, a·b refused), Hebrew geresh, Katakana middle dot, a mixed Arabic/Latin label refused by the Bidi rule, an A-label whose decoding would map differently refused, every length limit as before. All existing tests keep passing or change with a stated reason.
docs/design/DNS.md section 3 and slice 9 ("done"), docs/standard-library/dns.md, the doc comment of DomainName.tryFrom, docs/design/URI.md where it quotes the limitation; one paragraph in docs/PERFORMANCE.md that states the Unicode version and the measured size of all tables together in a native hello-world binary and in a program that uses every table (the old "size measured and stated" criterion of this work); torb docs skill docs skills regenerated.
Tier A and B green in forge CI; the registry and the HTTP client tests that build names (std/uri, std/http) still pass.

Seed impact

None new: no syntax, no new native (the previous issues made them known), no driver flag; std/dns names are not looked up by string. Nothing under seed/.

Size

One PR, two Conventional Commits: feat(dns): UTS #46 mapping, NFC, validity and the combining-mark rule and feat(dns): CONTEXTJ, CONTEXTO and the Bidi rule. If the diff of the first commit passes ~1500 changed lines excluding tests, the implementer splits the second commit into its own issue (label loop, depends on the first) and says so in the result comment.

## Context [DNS.md slice 9](https://git.torb.dev/torbscript/language/src/commit/393ef2608e4a60b8ff76d8b6b408951b822d887d/docs/design/DNS.md#L277): "the rest of UTS #46: the mapping table, NFC, the combining-mark, CONTEXTJ, CONTEXTO and Bidi rules". Today `DomainName.tryFrom` lower-cases with `Char.toLowerCase` and refuses what it cannot check; the table in section 3 ("What is not covered, exactly") says each row is "closed when [the table] does". This issue closes all of them. Last of the chain of #36. ## Depends on - #363 (`String.toNfc`) merged, and - #364 merged **and in a published seed**. Not `ready` before both. ## Decisions (refinement) - The processing stays what DNS.md section 3 says: UTS #46 lookup processing, non-transitional, `UseSTD3ASCIIRules=true`, `CheckHyphens=false`; **now also `CheckBidi=true` and `CheckJoiners=true`**, `VerifyDnsLength=true`. The section is rewritten to the as-built state; the "What is not covered" table is deleted and its rows move into the as-built text. - The step order of UTS #46 section 4: map (table, deviation characters `ß ς` kept, ZWJ/ZWNJ kept for CONTEXTJ), normalize to NFC, break at the dots, convert/validate each label (status check against the table, no combining mark first, hyphen rules as now, CONTEXTJ, CONTEXTO, Bidi rule per RFC 5893 on a domain that has a right-to-left label), Punycode, lengths. - **Behaviour changes are the documented plan**, not a breaking change of a promise: names with a disallowed code point that were encoded before are now refused with `DnsError.InvalidName` and a reason that names the code point (`U+XXXX`); upper-case non-Latin names are mapped (`ПРИМЕР.рф` becomes `xn--e1afmkfd.xn--p1ai`). `DomainName.ofPresentation` and `ofLabels` are untouched. - A reason string per refusal reuses the shape of `refusal` in `std/dns/src/name.trb`; no new `DnsError` case (the `DnsError` newtype question of RELEASE.md section 2 stays separate). - The old hand-made `mappedPoint`/`isIgnored`/`refusal` ranges are deleted, not kept as a fallback. ## Acceptance criteria - [ ] `DomainName.tryFrom` implements the steps above on top of `String.toNfc()` and the private natives of the IDNA issue; `std/dns/src/name.trb` loses the hand-written ranges. - [ ] **`IdnaTestV2.txt` of the pinned version passes in full** for the to-ASCII column with the settings above (cases the file marks as errors are refused, others give the expected A-labels): a std test runs a committed excerpt of at least 400 lines chosen to cover every status, every error class the file uses and each CONTEXTO rule; the whole file runs in CI after `tools/fetch-unicode.sh`. - [ ] Tests in `std/dns/tests/names.test.trb`: the Cyrillic and Greek upper-case names, `bu` + U+0308 equal to `bü`, a leading combining mark refused, `ß`/`ς` kept, a Persian name that needs ZWNJ accepted and a stray ZWNJ refused, the middle dot rule (`l·l` accepted, `a·b` refused), Hebrew geresh, Katakana middle dot, a mixed Arabic/Latin label refused by the Bidi rule, an A-label whose decoding would map differently refused, every length limit as before. All existing tests keep passing or change with a stated reason. - [ ] `docs/design/DNS.md` section 3 and slice 9 ("done"), `docs/standard-library/dns.md`, the doc comment of `DomainName.tryFrom`, `docs/design/URI.md` where it quotes the limitation; **one paragraph in `docs/PERFORMANCE.md` that states the Unicode version and the measured size of all tables together in a native hello-world binary and in a program that uses every table** (the old "size measured and stated" criterion of this work); `torb docs skill docs skills` regenerated. - [ ] Tier A and B green in forge CI; the registry and the HTTP client tests that build names (`std/uri`, `std/http`) still pass. ## Seed impact None new: no syntax, no new native (the previous issues made them known), no driver flag; `std/dns` names are not looked up by string. Nothing under `seed/`. ## Size One PR, two Conventional Commits: `feat(dns): UTS #46 mapping, NFC, validity and the combining-mark rule` and `feat(dns): CONTEXTJ, CONTEXTO and the Bidi rule`. If the diff of the first commit passes ~1500 changed lines excluding tests, the implementer splits the second commit into its own issue (label `loop`, depends on the first) and says so in the result comment.

orion added the

loop

label

2026-10-13 22:53:27 +00:00

orion referenced this issue

2026-10-13 22:53:36 +00:00

text: pinned Unicode data, generator and the character tables (slice 1 of the full Unicode tables) #36

orion referenced this issue

2026-10-13 22:53:41 +00:00

dns: IDNA, joining and bidi data and natives, teach the compiler (commit 1 of 2 for UTS #46) #364

orion referenced this issue

2026-10-13 22:53:57 +00:00