dns: IDNA, joining and bidi data and natives, teach the compiler (commit 1 of 2 for UTS #46)

orion commented

2026-10-13 22:53:26 +00:00

Collaborator

Context

Commit 1 of 2 for the IDNA data of std/dns (the use is #365). DNS.md section 3 lists what std/dns cannot do without tables: the full mapping (IdnaMappingTable.txt), the disallowed code points, the "starts with a combining mark" rule (general category), CONTEXTJ (joining type, virama), CONTEXTO (scripts Greek, Hebrew, Hiragana/Katakana/Han) and the Bidi rule (RFC 5893, bidi class). This issue brings that data, the runtime functions and the natives rows, and declares nothing in std/; the use is #365 and waits for a published seed (the Buffer precedent #299 / #54).

Depends on

#36 (generator, pinned data, fetch script). Independent of the normalization issues.

Decisions (refinement)

Private natives of std/dns, not public Char API: private native functions in std/dns (precedent: networkNameServers in std/network). A public Char.category / Char.bidiClass is a 1.0 commitment nobody has asked for yet; std/dns is the only consumer. If the natives manifest cannot name a native of a non-core package, the implementer puts them on a private type in std/text and reports it in the PR.
The natives, all Int in, small Int code out (no allocation): idnaStatus(point) (disallowed, ignored, mapped, deviation, valid, plus the disallowed_STD3_* variants), idnaMapping(point) (the mapped code points, as a short list appended to a caller's ArrayList<Int>), generalCategory(point), joiningType(point), bidiClass(point), canonicalCombiningClass(point) (virama is ccc 9), script(point) (only the scripts CONTEXTO names: Greek, Hebrew, Hiragana, Katakana, Han; Other for the rest).
Inputs added to the pinned set: IdnaMappingTable.txt (Public/17.0.0/idna/), extracted/DerivedJoiningType.txt, extracted/DerivedBidiClass.txt, Scripts.txt, IdnaTestV2.txt.
Table budget for all of this at most 60 KiB (the mapping table compresses as ranges plus a mapping pool); the implementer measures and states it.

Acceptance criteria

Generator and tools/fetch-unicode.sh extended with the inputs above and their SHA-256s; output reproducible as in #36.
The runtime functions behind the natives above and their ready rows in compiler/src/backend/c/natives.trb; header and machine_natives.c regenerated; VM and native agree.
A runtime C test that checks the status, mapping and class of every code point against the pinned data files when present (CI fetches), and a committed excerpt otherwise (all statuses, the longest mapping and one of each shorter length, deviation characters ß ς ZWJ ZWNJ, the CONTEXTO code points, an Arabic, a Hebrew, a Latin and a number code point per bidi class).
compiler/tests/natives.test.trb: each row exists and is ready.
Size, measured in forge CI and stated, with the same rule as #36 (4 KiB for a program that does not use them).
No change under std/, seed/, tests/conformance/. Tier A and B green. After the merge the seed is published (forge Actions -> seed); not part of this PR.

Seed impact

Yes: new natives (commit 1 of 2).

Size

One PR, two Conventional Commits: feat(text): IDNA, joining and bidi data in the generated Unicode tables and feat(runtime): the lookups of the IDNA data and their natives rows.

## Context Commit 1 of 2 for the IDNA data of `std/dns` (the use is #365). [DNS.md section 3](https://git.torb.dev/torbscript/language/src/commit/393ef2608e4a60b8ff76d8b6b408951b822d887d/docs/design/DNS.md#L119) lists what `std/dns` cannot do without tables: the full mapping (`IdnaMappingTable.txt`), the disallowed code points, the "starts with a combining mark" rule (general category), CONTEXTJ (joining type, virama), CONTEXTO (scripts Greek, Hebrew, Hiragana/Katakana/Han) and the Bidi rule (RFC 5893, bidi class). This issue brings that data, the runtime functions and the natives rows, and declares **nothing** in `std/`; the use is #365 and waits for a published seed (the `Buffer` precedent #299 / #54). ## Depends on #36 (generator, pinned data, fetch script). Independent of the normalization issues. ## Decisions (refinement) - **Private natives of `std/dns`**, not public `Char` API: `private native` functions in `std/dns` (precedent: `networkNameServers` in `std/network`). A public `Char.category` / `Char.bidiClass` is a 1.0 commitment nobody has asked for yet; `std/dns` is the only consumer. If the natives manifest cannot name a native of a non-core package, the implementer puts them on a private type in `std/text` and reports it in the PR. - The natives, all `Int` in, small `Int` code out (no allocation): `idnaStatus(point)` (disallowed, ignored, mapped, deviation, valid, plus the `disallowed_STD3_*` variants), `idnaMapping(point)` (the mapped code points, as a short list appended to a caller's `ArrayList<Int>`), `generalCategory(point)`, `joiningType(point)`, `bidiClass(point)`, `canonicalCombiningClass(point)` (virama is ccc 9), `script(point)` (only the scripts CONTEXTO names: Greek, Hebrew, Hiragana, Katakana, Han; `Other` for the rest). - Inputs added to the pinned set: `IdnaMappingTable.txt` (`Public/17.0.0/idna/`), `extracted/DerivedJoiningType.txt`, `extracted/DerivedBidiClass.txt`, `Scripts.txt`, `IdnaTestV2.txt`. - Table budget for all of this at most **60 KiB** (the mapping table compresses as ranges plus a mapping pool); the implementer measures and states it. ## Acceptance criteria - [ ] Generator and `tools/fetch-unicode.sh` extended with the inputs above and their SHA-256s; output reproducible as in #36. - [ ] The runtime functions behind the natives above and their ready rows in `compiler/src/backend/c/natives.trb`; header and `machine_natives.c` regenerated; VM and native agree. - [ ] A runtime C test that checks the status, mapping and class of every code point against the pinned data files when present (CI fetches), and a committed excerpt otherwise (all statuses, the longest mapping and one of each shorter length, deviation characters `ß ς ZWJ ZWNJ`, the CONTEXTO code points, an Arabic, a Hebrew, a Latin and a number code point per bidi class). - [ ] `compiler/tests/natives.test.trb`: each row exists and is ready. - [ ] **Size, measured in forge CI and stated**, with the same rule as #36 (4 KiB for a program that does not use them). - [ ] No change under `std/`, `seed/`, `tests/conformance/`. Tier A and B green. After the merge the seed is published (forge Actions -> `seed`); **not part of this PR**. ## Seed impact Yes: new natives (commit 1 of 2). ## Size One PR, two Conventional Commits: `feat(text): IDNA, joining and bidi data in the generated Unicode tables` and `feat(runtime): the lookups of the IDNA data and their natives rows`.

orion added the

loop

label

2026-10-13 22:53:26 +00:00

orion referenced this issue

2026-10-13 22:53:36 +00:00

text: pinned Unicode data, generator and the character tables (slice 1 of the full Unicode tables) #36

orion referenced this issue

2026-10-13 22:53:41 +00:00

dns: the rest of UTS #46 - mapping table, NFC, CONTEXTJ/CONTEXTO and Bidi (DNS.md slice 9) #365

orion referenced this issue

2026-10-13 22:53:57 +00:00

text: pinned Unicode data, generator and the character tables (slice 1 of the full Unicode tables) #36

orion commented

2026-10-13 22:53:59 +00:00

Author

Collaborator

[loop:torbscript] refine: body written by the refinement of #36 (goal, criteria, decisions, seed impact, size are in it). next: waiting (depends on #36 being merged)

orion referenced this issue

2026-10-13 22:53:59 +00:00

dns: the rest of UTS #46 - mapping table, NFC, CONTEXTJ/CONTEXTO and Bidi (DNS.md slice 9) #365

orion referenced this issue

2026-10-14 23:52:22 +00:00

text: pinned Unicode data, generator and the character tables (slice 1 of the full Unicode tables) #36

orion added the

ready

label

2026-10-15 22:03:14 +00:00

orion commented

2026-10-15 22:03:14 +00:00

Author

Collaborator

[loop:torbscript] decided by the loop: the only dependency, #36, has landed (commit c3eee74a31); body already states goal, criteria, seed impact (new natives, commit 1 of 2) and size. Definition of ready met, labelled ready.

[loop:torbscript] claim implement by session:loop-torbscript-20261016 until 2026-10-16T04:03Z

[loop:torbscript] decided by the loop: the only dependency, #36, has landed (commit c3eee74a3123d9cd8030449fbc076666b1d93d23); body already states goal, criteria, seed impact (new natives, commit 1 of 2) and size. Definition of ready met, labelled `ready`. [loop:torbscript] claim implement by session:loop-torbscript-20261016 until 2026-10-16T04:03Z

orion referenced this issue

2026-10-15 22:08:24 +00:00

text: Unicode case folding data and the natives for Char.category and case folding (teach the compiler, commit 1 of 2 for #366) #372

orion referenced this issue

2026-10-15 22:08:31 +00:00

regex: Unicode categories for \p{...} and case folding from the tables (slice of #366) #374

orion referenced this issue

2026-10-15 22:08:44 +00:00

text: public Char.category and the Unicode tables for regex and markdown (follow-up of #36) #366

orion referenced this issue from a pull request that will close it,

2026-10-15 22:51:45 +00:00

feat: IDNA, joining and bidi data and natives (commit 1 of 2 for UTS #46) #376

argus closed this issue

2026-10-15 23:13:33 +00:00

orion commented

2026-10-15 23:14:33 +00:00

Author

Collaborator

[loop:torbscript] implement: #376 opened (two commits, 17bd8da6 data and ab769db4 natives; tier A, tier B, bootstrap, every-host-same-C, gates and pdisc green on the head), reviewed and merged by Argus, next: waiting (commit 2 of 2 for UTS #46, which builds on these natives).

Measured in forge CI (tier B, tools/size-cost.sh, head ab769db4):

Table: 60,004 bytes (budget 61,440): top 2,176 + middle 6,976 + leaves 14,160 + records 1,536 + entries 30,016 + pool 5,140. Object idna_data.o 60,064 bytes, idna.o 1,477 bytes.
A program that does not use the data: hello-world, native --release, 376,456 -> 376,792 bytes (+336; bound 4,096). The idna parts are linked only when a program calls one of the seven natives.
Excerpt: runtime/tests/idna-excerpt.txt, 284 code points, always checked; the full 0x110000 check runs where CI has fetched the pinned files.

Notes:

Unicode 17.0.0 UTS #46 has no disallowed_STD3_* statuses, so the generator accepts the five that exist (disallowed, ignored, mapped, deviation, valid). The issue text mentions the STD3 ones.
Left over: runtime/README.md "The IDNA tables" has no measured size table yet (the numbers above came back after the merge). Fold it into the next commit.

Worker tokens: not measurable from inside the session; two context windows, roughly 150 tool calls.

[loop:torbscript] implement: https://git.torb.dev/torbscript/language/pulls/376 opened (two commits, 17bd8da6 data and ab769db4 natives; tier A, tier B, bootstrap, every-host-same-C, gates and pdisc green on the head), reviewed and merged by Argus, next: waiting (commit 2 of 2 for UTS #46, which builds on these natives). Measured in forge CI (tier B, `tools/size-cost.sh`, head ab769db4): - Table: 60,004 bytes (budget 61,440): top 2,176 + middle 6,976 + leaves 14,160 + records 1,536 + entries 30,016 + pool 5,140. Object `idna_data.o` 60,064 bytes, `idna.o` 1,477 bytes. - A program that does not use the data: hello-world, native --release, 376,456 -> 376,792 bytes (+336; bound 4,096). The idna parts are linked only when a program calls one of the seven natives. - Excerpt: `runtime/tests/idna-excerpt.txt`, 284 code points, always checked; the full 0x110000 check runs where CI has fetched the pinned files. Notes: - Unicode 17.0.0 UTS #46 has no `disallowed_STD3_*` statuses, so the generator accepts the five that exist (disallowed, ignored, mapped, deviation, valid). The issue text mentions the STD3 ones. - Left over: `runtime/README.md` "The IDNA tables" has no measured size table yet (the numbers above came back after the merge). Fold it into the next commit. Worker tokens: not measurable from inside the session; two context windows, roughly 150 tool calls.

orion referenced this issue

2026-10-16 22:30:40 +00:00

feat(text): Unicode case folding data and the natives for Char.category and case folding #378

orion referenced this issue

2026-10-16 23:06:37 +00:00

text: Unicode case folding data and the natives for Char.category and case folding (teach the compiler, commit 1 of 2 for #366) #372