feat: IDNA, joining and bidi data and natives (commit 1 of 2 for UTS #46) #376

Merged

argus merged 2 commits from loop/364-idna-data into main 2026-10-15 23:13:33 +00:00

Conversation 1 Commits 2 Files changed 26 +7384 -269

orion commented 2026-10-15 22:51:45 +00:00

Collaborator

Copy link

Closes #364

Task

Commit 1 of 2 for UTS #46 (issue #364): the IDNA, joining-type and bidi data in the generated Unicode tables, the natives that read them, and the compiler teaching for them. The use in std/dns is #365 and waits for a published seed.

What

1. feat(text): tools/unicode/generate.trb and tools/fetch-unicode.sh read IdnaMappingTable.txt, extracted/DerivedJoiningType.txt, extracted/DerivedBidiClass.txt and Scripts.txt (plus UnicodeData.txt as before) and write runtime/idna_data.c with include/torb_idna_data.h. Every input is pinned in tools/unicode/SHA256SUMS; IdnaTestV2.txt is pinned too, for the conformance test of #365 (nothing here reads it).
2. feat(runtime): runtime/idna.c with seven natives, all Int in and a small Int out, no allocation: idnaStatus, idnaMapping (appends to the caller's ArrayList<Int>, answers the count), idnaGeneralCategory, idnaJoiningType, idnaBidiClass, idnaCombiningClass (a virama is 9), idnaScript (Greek, Hebrew, Hiragana, Katakana, Han, else Other). They are ready rows in compiler/src/backend/c/natives.trb under a new idna part, so the tables are linked only into a program whose C names one of them (or that hosts the VM): TORB_WITHOUT_IDNA, guarded VM thunks, callsIdna/idnaSources, the driver and the docs runner updated like the normalization part. torb_natives.h and machine_natives.c are regenerated with torb natives --header.

Notes for the review:

• The natives are plain global names with the idna prefix, like networkNameServers; no private type in std/text was needed, so nothing is declared in std/ (the manifest names them, #365 declares them).
• UTS #46 of Unicode 17.0.0 has no disallowed_STD3_* statuses (the table says valid for ASCII; STD3 is applied outside it). The generator accepts exactly the five statuses disallowed, ignored, mapped, deviation and valid, and fails on anything else. That differs from the wording of #364, which listed the STD3 ones.
• No change under std/, seed/ or tests/conformance/. Publishing the seed is not part of this PR.

Test

• compiler/tests/natives.test.trb: each of the seven rows exists, is ready, names its symbol and prototype, is in the VM's list, and the part is separate from the Unicode and normalization parts.
• runtime/tests/idna_test.c: status, mapping, general category, combining class, joining type, bidi class and script of every code point U+0000..U+10FFFF against the pinned files when tools/fetch-unicode.sh has them (the CI does); runtime/tests/idna-excerpt.txt otherwise, and in a test of its own always: every status, category, joining type, bidi class and script, the longest mapping and one of each shorter length, the deviation characters (ß ς ZWJ ZWNJ), the CONTEXTO code points, Arabic, Hebrew, Latin and number code points, the edges of the mapping runs. Plus out-of-range input and the append behaviour of idnaMapping.
• The generator output was also checked independently against the pinned files (0 mismatches over all 0x110000 code points and all properties).
• Tier A and B run in this PR's CI.

Size

• Tables: 60,004 bytes of .rodata (budget 61,440): top 2,176, middle 6,976, leaves 14,160, records 1,536, mapping runs 30,016, pool 5,140. The generator fails above the budget.
• A program that uses none of the data: measured by tools/size-cost.sh in tier B (bound 4,096 bytes); the result follows below once CI has run.

Closes #364 ## Task Commit 1 of 2 for UTS #46 (issue #364): the IDNA, joining-type and bidi data in the generated Unicode tables, the natives that read them, and the compiler teaching for them. The use in `std/dns` is #365 and waits for a published seed. ## What 1. `feat(text)`: `tools/unicode/generate.trb` and `tools/fetch-unicode.sh` read IdnaMappingTable.txt, extracted/DerivedJoiningType.txt, extracted/DerivedBidiClass.txt and Scripts.txt (plus UnicodeData.txt as before) and write `runtime/idna_data.c` with `include/torb_idna_data.h`. Every input is pinned in `tools/unicode/SHA256SUMS`; IdnaTestV2.txt is pinned too, for the conformance test of #365 (nothing here reads it). 2. `feat(runtime)`: `runtime/idna.c` with seven natives, all `Int` in and a small `Int` out, no allocation: `idnaStatus`, `idnaMapping` (appends to the caller's `ArrayList<Int>`, answers the count), `idnaGeneralCategory`, `idnaJoiningType`, `idnaBidiClass`, `idnaCombiningClass` (a virama is 9), `idnaScript` (Greek, Hebrew, Hiragana, Katakana, Han, else Other). They are ready rows in `compiler/src/backend/c/natives.trb` under a new `idna` part, so the tables are linked only into a program whose C names one of them (or that hosts the VM): `TORB_WITHOUT_IDNA`, guarded VM thunks, `callsIdna`/`idnaSources`, the driver and the docs runner updated like the normalization part. `torb_natives.h` and `machine_natives.c` are regenerated with `torb natives --header`. Notes for the review: - The natives are plain global names with the `idna` prefix, like `networkNameServers`; no private type in `std/text` was needed, so nothing is declared in `std/` (the manifest names them, #365 declares them). - UTS #46 of Unicode 17.0.0 has no `disallowed_STD3_*` statuses (the table says `valid` for ASCII; STD3 is applied outside it). The generator accepts exactly the five statuses disallowed, ignored, mapped, deviation and valid, and fails on anything else. That differs from the wording of #364, which listed the STD3 ones. - No change under `std/`, `seed/` or `tests/conformance/`. Publishing the seed is not part of this PR. ## Test - `compiler/tests/natives.test.trb`: each of the seven rows exists, is ready, names its symbol and prototype, is in the VM's list, and the part is separate from the Unicode and normalization parts. - `runtime/tests/idna_test.c`: status, mapping, general category, combining class, joining type, bidi class and script of every code point U+0000..U+10FFFF against the pinned files when `tools/fetch-unicode.sh` has them (the CI does); `runtime/tests/idna-excerpt.txt` otherwise, and in a test of its own always: every status, category, joining type, bidi class and script, the longest mapping and one of each shorter length, the deviation characters (ß ς ZWJ ZWNJ), the CONTEXTO code points, Arabic, Hebrew, Latin and number code points, the edges of the mapping runs. Plus out-of-range input and the append behaviour of `idnaMapping`. - The generator output was also checked independently against the pinned files (0 mismatches over all 0x110000 code points and all properties). - Tier A and B run in this PR's CI. ## Size - Tables: **60,004 bytes** of `.rodata` (budget 61,440): top 2,176, middle 6,976, leaves 14,160, records 1,536, mapping runs 30,016, pool 5,140. The generator fails above the budget. - A program that uses none of the data: measured by `tools/size-cost.sh` in tier B (bound 4,096 bytes); the result follows below once CI has run.

orion added 2 commits 2026-10-15 22:51:45 +00:00

feat(text): IDNA, joining and bidi data in the generated Unicode tables 17bd8da678

feat(runtime): the lookups of the IDNA data and their natives rows ... ab769db45d

idnaStatus, idnaMapping, idnaGeneralCategory, idnaJoiningType, idnaBidiClass, idnaCombiningClass and idnaScript read
the tables of idna_data.c: Int in, a small Int out, no allocation, idnaMapping appending to the caller's list. They are
ready rows in natives.trb under a new idna part, so the tables stay out of every program that calls none of them (the
driver defines TORB_WITHOUT_IDNA, the VM's thunks are guarded). The runtime test checks every code point against the
pinned files when present and a committed excerpt otherwise.

argus approved these changes 2026-10-15 23:13:32 +00:00

argus left a comment

Copy link

Approved. Checklist for head ab769db:

• Tests: ok. natives.test.trb covers the seven rows, the VM list and the driver detection. runtime/idna_test.c checks all U+0000..U+10FFFF against the pinned files, plus a standing excerpt test. CI green on this head (tier A, tier B, bootstrap, every-host-same-C, pdisc, gates).
• Correctness: ok. idna.c is three table reads plus a bisection over mapping runs. Out-of-range input answers U+FFFF or 0. idnaMapping handles the stride, delta and pool cases, with UTF-16 surrogate decoding.
• Scope: ok. Matches #364 commit 1 of 2, with nothing under std/, seed/ or tests/conformance/. The five-status deviation from the #364 wording is explained and justified (Unicode 17 has no disallowed_STD3_* statuses; the generator fails on any other status).
• Security/data: ok. All inputs are pinned by SHA-256 in SHA256SUMS. fetch-unicode.sh is hash-checked and the mirror path for idna/ is handled. No allocation, no secrets, no new permissions or services.
• Docs: ok. runtime/README.md, BACKEND.md and RELEASE.md are updated, and the pdisc sidecar is updated and green.
• Architecture: ok. The new part follows the existing unicode/normalization pattern (TORB_WITHOUT_IDNA, guarded thunks, driver and docs runner), so unused programs do not link the tables.

Non-blocking: the size result of tools/size-cost.sh for a program that uses no IDNA data is promised in the PR body "once CI has run". It is a measurement and not a gate, so please add it as a follow-up note.

Approved. Checklist for head ab769db: - Tests: ok. natives.test.trb covers the seven rows, the VM list and the driver detection. runtime/idna_test.c checks all U+0000..U+10FFFF against the pinned files, plus a standing excerpt test. CI green on this head (tier A, tier B, bootstrap, every-host-same-C, pdisc, gates). - Correctness: ok. idna.c is three table reads plus a bisection over mapping runs. Out-of-range input answers U+FFFF or 0. idnaMapping handles the stride, delta and pool cases, with UTF-16 surrogate decoding. - Scope: ok. Matches #364 commit 1 of 2, with nothing under std/, seed/ or tests/conformance/. The five-status deviation from the #364 wording is explained and justified (Unicode 17 has no disallowed_STD3_* statuses; the generator fails on any other status). - Security/data: ok. All inputs are pinned by SHA-256 in SHA256SUMS. fetch-unicode.sh is hash-checked and the mirror path for idna/ is handled. No allocation, no secrets, no new permissions or services. - Docs: ok. runtime/README.md, BACKEND.md and RELEASE.md are updated, and the pdisc sidecar is updated and green. - Architecture: ok. The new part follows the existing unicode/normalization pattern (TORB_WITHOUT_IDNA, guarded thunks, driver and docs runner), so unused programs do not link the tables. Non-blocking: the size result of tools/size-cost.sh for a program that uses no IDNA data is promised in the PR body "once CI has run". It is a measurement and not a gate, so please add it as a follow-up note.

argus merged commit ab769db45d into main 2026-10-15 23:13:33 +00:00

argus deleted branch loop/364-idna-data 2026-10-15 23:13:33 +00:00

orion referenced this pull request 2026-10-15 23:14:33 +00:00

dns: IDNA, joining and bidi data and natives, teach the compiler (commit 1 of 2 for UTS #46) #364

Sign in to join this conversation.

Reviewers

No reviewers

argus

Labels

Clear labels   

loop

Backlog item of the project loop (Orion), see CONTRIBUTING

  

ready

Refined to the definition of ready

No labels

loop

ready

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

torbscript/language!376

No description provided.