feat(text): generate the Unicode tables and read them in the case natives #368

Merged

argus merged 2 commits from loop/36-unicode-tables into main 2026-10-14 23:50:42 +00:00

Conversation 1 Commits 2 Files changed 34 +4017 -185

orion commented 2026-10-14 22:40:48 +00:00

Collaborator

Copy link

Closes #36

Slice 1 of the full Unicode tables: pinned data, a reproducible generator, and the six natives reading the tables.

What changes

• tools/fetch-unicode.sh downloads the pinned Unicode 17.0.0 files and verifies them against tools/unicode/SHA256SUMS (kept in build/unicode/, not committed).
• tools/unicode/generate.trb writes runtime/unicode_data.c and runtime/include/torb_unicode_data.h. Running it twice gives byte-identical files.
• runtime/unicode.c: Char.isLetter, Char.isWhitespace, Char.toUpperCase, Char.toLowerCase (simple mappings), String.toUpperCase, String.toLowerCase (full default conversion: unconditional SpecialCasing mappings plus the Final_Sigma rule, no locale).
• The VM and the native back end share these natives, so they agree. tests/conformance/character-case.trb is rewritten and extended; the native build output matches the expected file.
• Tests: runtime/tests/unicode_test.c checks the generated tables for every code point against UnicodeData.txt when present (runtime/build.sh fetches it, best effort) and against the committed excerpt runtime/tests/unicode-excerpt.txt otherwise. text_test.c runs the same check through the natives, plus White_Space for every code point, special casing and Final_Sigma.
• Lexer: names with letters are unchanged (tests/lexer-cases passes as is). One diagnostic changes with the exact isLetter: a digit of another script (count٣) was a "letter" by the old approximation and is now an unexpected character; compiler/tests/lexer.test.trb is updated with it.
• tests/conformance/character-case.allocations goes from 90 to 270 blocks: the extended program converts and compares many more texts.
• Docs: the six native doc comments, the two "until milestone 8" bullets in CONCEPT.md, docs/standard-library/text.md, docs/BACKEND.md, docs/design/RELEASE.md, runtime/README.md, and the regenerated skills/. docs/design/TEXT-FORMATS.md is untouched.

Deviations from the issue

1. Fourth pinned input: DerivedCoreProperties.txt. Cased and Case_Ignorable are derived properties, needed by Final_Sigma. Its hash is pinned with the other three.
2. Three-level table (4 + 5 + 12 bits) instead of the two-level layout: top 2176 entries, middle 3296, leaves 12128, 223 records of 12 bytes. Total table object ~25.8 KiB.
3. Linked on demand. The tables would have grown every program by about 27 KiB, so the six natives are in a manifest part unicode. The driver links unicode.c and unicode_data.c only when the emitted C names one of the six (or the program hosts the VM, which calls every native); otherwise it defines TORB_WITHOUT_UNICODE, and the generated thunks panic with "not linked into this program" (same pattern as TLS). A seed driver that links all *.c and defines no macro keeps working. Nothing under seed/, no syntax change, no new native.

Size (hello world, --release, native binary)

	without section GC	with lld section GC
before	369248 B	214208 B
after, tables not linked	368080 B (−1168)	212768 B
after, tables linked	397136 B (+27888)	212840 B

unicode_data.o: 26456 B rodata; unicode.o ~2.1 KiB. A program that never uses a table-backed native does not grow (limit 4 KiB).

Verification

• Quick runtime test harness (zig cc, -Wall -Wextra -Wpedantic -Werror): 36 tests, 0 failed, both with the whole UnicodeData.txt and with the excerpt.
• Forge CI is green on the pushed head (tier A, tier B, playground, pdisc).
• torb test compiler/tests/natives.test.trb: 18 passed. torb format --check clean on the touched files; torb docs skill docs skills --check up to date.

Closes #36 Slice 1 of the full Unicode tables: pinned data, a reproducible generator, and the six natives reading the tables. ## What changes - `tools/fetch-unicode.sh` downloads the pinned Unicode 17.0.0 files and verifies them against `tools/unicode/SHA256SUMS` (kept in `build/unicode/`, not committed). - `tools/unicode/generate.trb` writes `runtime/unicode_data.c` and `runtime/include/torb_unicode_data.h`. Running it twice gives byte-identical files. - `runtime/unicode.c`: `Char.isLetter`, `Char.isWhitespace`, `Char.toUpperCase`, `Char.toLowerCase` (simple mappings), `String.toUpperCase`, `String.toLowerCase` (full default conversion: unconditional SpecialCasing mappings plus the Final_Sigma rule, no locale). - The VM and the native back end share these natives, so they agree. `tests/conformance/character-case.trb` is rewritten and extended; the native build output matches the expected file. - Tests: `runtime/tests/unicode_test.c` checks the generated tables for every code point against `UnicodeData.txt` when present (`runtime/build.sh` fetches it, best effort) and against the committed excerpt `runtime/tests/unicode-excerpt.txt` otherwise. `text_test.c` runs the same check through the natives, plus White_Space for every code point, special casing and Final_Sigma. - Lexer: names with letters are unchanged (`tests/lexer-cases` passes as is). One diagnostic changes with the exact `isLetter`: a digit of another script (`count٣`) was a "letter" by the old approximation and is now an unexpected character; `compiler/tests/lexer.test.trb` is updated with it. - `tests/conformance/character-case.allocations` goes from 90 to 270 blocks: the extended program converts and compares many more texts. - Docs: the six native doc comments, the two "until milestone 8" bullets in CONCEPT.md, `docs/standard-library/text.md`, `docs/BACKEND.md`, `docs/design/RELEASE.md`, `runtime/README.md`, and the regenerated `skills/`. `docs/design/TEXT-FORMATS.md` is untouched. ## Deviations from the issue 1. **Fourth pinned input: `DerivedCoreProperties.txt`.** Cased and Case_Ignorable are derived properties, needed by Final_Sigma. Its hash is pinned with the other three. 2. **Three-level table (4 + 5 + 12 bits)** instead of the two-level layout: `top` 2176 entries, `middle` 3296, `leaves` 12128, 223 records of 12 bytes. Total table object ~25.8 KiB. 3. **Linked on demand.** The tables would have grown every program by about 27 KiB, so the six natives are in a manifest part `unicode`. The driver links `unicode.c` and `unicode_data.c` only when the emitted C names one of the six (or the program hosts the VM, which calls every native); otherwise it defines `TORB_WITHOUT_UNICODE`, and the generated thunks panic with "not linked into this program" (same pattern as TLS). A seed driver that links all `*.c` and defines no macro keeps working. Nothing under `seed/`, no syntax change, no new native. ## Size (hello world, `--release`, native binary) | | without section GC | with lld section GC | |---|---|---| | before | 369248 B | 214208 B | | after, tables not linked | 368080 B (−1168) | 212768 B | | after, tables linked | 397136 B (+27888) | 212840 B | `unicode_data.o`: 26456 B rodata; `unicode.o` ~2.1 KiB. A program that never uses a table-backed native does not grow (limit 4 KiB). ## Verification - Quick runtime test harness (zig cc, `-Wall -Wextra -Wpedantic -Werror`): 36 tests, 0 failed, both with the whole `UnicodeData.txt` and with the excerpt. - Forge CI is green on the pushed head (tier A, tier B, playground, pdisc). - `torb test compiler/tests/natives.test.trb`: 18 passed. `torb format --check` clean on the touched files; `torb docs skill docs skills --check` up to date.

orion added 2 commits 2026-10-14 22:40:48 +00:00

feat(text): generate the Unicode tables from the pinned character database ... 241ca4c8a8

tools/fetch-unicode.sh downloads the pinned files of Unicode 17.0.0 and checks
them against tools/unicode/SHA256SUMS. tools/unicode/generate.trb reads
UnicodeData.txt, SpecialCasing.txt, PropList.txt and DerivedCoreProperties.txt
and writes runtime/unicode_data.c and its header: a three-level table of 223
records (general category, simple case deltas, White_Space, Cased,
Case_Ignorable) and the 103 unconditional full mappings.

DerivedCoreProperties.txt is a fourth pinned input: Cased and Case_Ignorable
are derived properties that the Final_Sigma rule needs.

A runtime test compares the tables with every code point of UnicodeData.txt
when the pinned file is present (runtime/build.sh fetches it), and with a
committed excerpt otherwise.

feat(text): case mapping and letters from the full tables ... a96ba0b77d

Char.isLetter, Char.isWhitespace, Char.toUpperCase, Char.toLowerCase,
String.toUpperCase and String.toLowerCase read the generated tables
(runtime/unicode.c): the simple mappings of UnicodeData.txt for a Char, the
full default conversion of the Unicode core specification for a String, with
the unconditional SpecialCasing.txt mappings and the Final_Sigma rule.

The tables are linked on demand. The six natives belong to the "unicode" part
of the manifest, and the driver links unicode.c and unicode_data.c only into a
program whose C names one of them (or that hosts the VM); every other program
is built with TORB_WITHOUT_UNICODE and carries a panicking thunk.

tests/conformance/character-case.trb is rewritten and extended. The doc
comments of the six natives, CONCEPT.md, the standard library text page and the
generated skills are updated.

orion requested review from argus 2026-10-14 22:40:49 +00:00

orion force-pushed loop/36-unicode-tables from a96ba0b77d to 0f81fc19b7 2026-10-14 22:43:46 +00:00 Compare

orion force-pushed loop/36-unicode-tables from 0f81fc19b7 to 3f4da233f2 2026-10-14 23:00:30 +00:00 Compare

orion force-pushed loop/36-unicode-tables from 3f4da233f2 to c3eee74a31 2026-10-14 23:23:23 +00:00 Compare

orion referenced this pull request 2026-10-14 23:50:33 +00:00

text: pinned Unicode data, generator and the character tables (slice 1 of the full Unicode tables) #36

argus approved these changes 2026-10-14 23:50:40 +00:00

argus left a comment

Copy link

Approved at c3eee74. Checklist:

• Tests: ok. New runtime tests (unicode_test.c, text_test.c) check every code point against UnicodeData.txt or the committed excerpt, plus White_Space, special casing and Final_Sigma; natives.test.trb covers the part/driver detection; conformance character-case extended. CI (gates, bootstrap, tier A/B, same-C, playground, pdisc) is green on this head; the other-platform jobs are skipped by the workflow, not failed.
• Correctness: ok. Three-level lookup is bounds-safe (code points above U+10FFFF map to the unassigned record), the two-pass conversion sizes the result before writing (overflow guarded), Final_Sigma scans both directions and stops on invalid bytes, and invalid bytes are still copied through.
• Supply chain/security: ok. The four SHA-256 pins in tools/unicode/SHA256SUMS match the files at unicode.org for 17.0.0 (I fetched and hashed them); the fetch script never keeps a download with another hash, and the input is validated as digits and dots.
• Scope: ok. Matches #36 with the three deviations declared in the body; no seed, syntax or new-native change.
• Docs: ok (native doc comments, CONCEPT, text.md, BACKEND, runtime README, regenerated skills; pdisc green).
• Behaviour change noted and documented: String.toUpperCase can now be longer than its input.

Non-blocking: none.

Approved at c3eee74. Checklist: - Tests: ok. New runtime tests (`unicode_test.c`, `text_test.c`) check every code point against `UnicodeData.txt` or the committed excerpt, plus White_Space, special casing and Final_Sigma; `natives.test.trb` covers the part/driver detection; conformance `character-case` extended. CI (gates, bootstrap, tier A/B, same-C, playground, pdisc) is green on this head; the other-platform jobs are skipped by the workflow, not failed. - Correctness: ok. Three-level lookup is bounds-safe (code points above U+10FFFF map to the unassigned record), the two-pass conversion sizes the result before writing (overflow guarded), Final_Sigma scans both directions and stops on invalid bytes, and invalid bytes are still copied through. - Supply chain/security: ok. The four SHA-256 pins in `tools/unicode/SHA256SUMS` match the files at unicode.org for 17.0.0 (I fetched and hashed them); the fetch script never keeps a download with another hash, and the input is validated as digits and dots. - Scope: ok. Matches #36 with the three deviations declared in the body; no seed, syntax or new-native change. - Docs: ok (native doc comments, CONCEPT, text.md, BACKEND, runtime README, regenerated skills; pdisc green). - Behaviour change noted and documented: `String.toUpperCase` can now be longer than its input. Non-blocking: none.

argus merged commit c3eee74a31 into main 2026-10-14 23:50:42 +00:00

argus deleted branch loop/36-unicode-tables 2026-10-14 23:50:42 +00:00

orion referenced this pull request 2026-10-14 23:52:22 +00:00

text: pinned Unicode data, generator and the character tables (slice 1 of the full Unicode tables) #36

Sign in to join this conversation.

Reviewers

No reviewers

argus

Labels

Clear labels   

loop

Backlog item of the project loop (Orion), see CONTRIBUTING

  

ready

Refined to the definition of ready

No labels

loop

ready

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

torbscript/language!368

No description provided.