text: pinned Unicode data, generator and the character tables (slice 1 of the full Unicode tables) #36
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Context
std/textanswersChar.isLetter,Char.isWhitespace,Char.toUpperCase,Char.toLowerCaseandString.toUpperCase/toLowerCasefrom a small closed table: ASCII, the letters of Latin-1 andÿ/Ÿ(runtime/text.c,torb_char_to_upper_caseand friends;isLetterabove Latin-1 is "everything"). RELEASE.md section 2 ("1.0") needs the full tables: a name thatstddeclares has to work. This is slice 1 of 5 of the Unicode work (chain below); it brings the data and the generator, and switches the natives that already exist over to it. No new native, no syntax, no seed change.Source
The chain (dependency order)
String.toNfc()and friends instd/text(commit 2 of 2) - depends on #362 and a published seed containing it.std/dns, teach the compiler (commit 1 of 2) - depends on this issue.std/dns- depends on #363 and #364 (and a published seed for 4).Follow-up outside the chain: #366 (public
Char.category, regex and markdown tables).Decisions (made in refinement, see the comment "decided by the loop")
runtime/(one new generated translation unit,runtime/unicode_data.c, and its header part), used by the native back end and the VM through the same functions (the VM thunks come from the natives manifest as today). Not TorbScript tables: ten thousand element literals cost compile time in every program that importsstd/text, and two implementations could drift.tools/unicode/generate.trb(TorbScript, run withtorb run), inputsUnicodeData.txt,SpecialCasing.txt,PropList.txtof the pinned version. Inputs are not committed:tools/fetch-unicode.sh <version>downloads them tobuild/unicode/<version>/and checks a SHA-256 list kept intools/unicode/SHA256SUMS. The generated file is committed with a header (version, input hashes, "generated, do not edit") and markedlinguist-generatedin.gitattributes. Reviewing means the generator and the tests, not the array.0x110000 / blockrange; stage 2: deduplicated blocks of packed records). Block size chosen by the generator by trying 32, 64, 128, 256 and keeping the smallest total; record holds the general category (5 bits), the simple upper and lower mapping as a signed delta, and theWhite_Spacebit.Char.toUpperCase/toLowerCasestay the simple one-to-one mapping (UnicodeData.txt), now complete:'α'.toUpperCase() == 'Α','ß'.toUpperCase() == 'ß'(its upper case is two code points),'ÿ'/'Ÿ'as before.String.toUpperCase/toLowerCasebecome the full default case conversion (Unicode core specification 3.13): unconditionalSpecialCasing.txtmappings ("ß".toUpperCase() == "SS","ʼn","ǰ", ligatures) andFinal_Sigmafor lower case ("ΑΣ".toLowerCase() == "ας"). The result can be longer than the input. No language-tailored mappings (Turkish, Azeri, Lithuanian) and no locale parameter.Char.isLetteris exact: general categoryL*.Char.isDigitis unchanged (ASCII0to9, as its doc comment says).Char.isWhitespaceis theWhite_Spaceproperty (what the current list approximates).runtime/README.md.Acceptance criteria
tools/fetch-unicode.shdownloads the pinned inputs and fails on a hash mismatch;tools/unicode/generate.trbwritesruntime/unicode_data.c(and the declarations it needs inruntime/include/) reproducibly: running it twice, and on Linux and Windows, gives byte-identical output (CRLF rules of the repo kept for the generated file).torb_char_is_letter,torb_char_is_whitespace,torb_char_to_upper_case,torb_char_to_lower_case,torb_text_to_upper_caseandtorb_text_to_lower_caseread the tables. Both back ends agree: the conformance suite runs in the VM and natively.isLetterequals "general category starts with L" and the simple mappings equalUnicodeData.txt- checked by a runtime C test (runtime/tests/text_test.c) that reads the pinnedUnicodeData.txtwhen present (CI fetches it) and by a committed excerpt otherwise.tests/conformance/character-case.trbis rewritten for the new rules and extended: Greek, Cyrillic, Armenian, Georgian (Mtavrulipair), Deseret (plane 1), aß/ʼn/ǰ/ffifull-mapping row,Final_Sigmaat the end of a word and before a letter, an empty string, and'ß'.toUpperCase() == 'ß'.compiler/src/syntax/lexer.trb(isWordStart/isWordPartuseisLetter) tokenises ASCII and non-ASCII input exactly as before, or the diagnostic that changes is updated deliberately with its test intests/lexer-cases.sh tools/gates.sh aandbare green in forge CI, the bootstrap fixpoint holds.size/nm --size-sort). Rule: a program that never uses a table-backed native must not grow by more than 4 KiB; if the plain link exceeds that, the tables go into an object that is linked only when referenced (section GC or an archive), decided by the measured delta and recorded in a comment. The numbers go intoruntime/README.mdand the PR body.docs/design/TEXT-FORMATS.mdis not changed (regex and markdown tables are not in this chain),docs/standard-library/text page;torb docs skill docs skillsregenerated and committed.Seed impact
None expected: no syntax change, no new native (the natives manifest and
machine_natives.cstay as they are), no driver flag, no std name looked up by string. Existing natives change behaviour above Latin-1 only. The bootstrap fixpoint compare is the proof; if it differs, the implementer reports instead of refreshing anything. Nothing underseed/.Size
One PR, two Conventional Commits:
feat(text): generate the Unicode tables from the pinned character database(fetch script, generator, generated data, runtime test of the data) andfeat(text): case mapping and letters from the full tables(the six natives, conformance, docs).Out of scope
Normalization (issues of the chain), IDNA and the DNS rules (chain), a public
Char.category, replacing the hand-made tables ofstd/regexandstd/markdown(a follow-up that needs a public category API and its own design), locale-tailored case mapping.Moved from the planning lists of the docs; the design record stays the specification.
[loop:torbscript] intake: in scope (author orion, label loop). Full Unicode tables for
std/text(ROADMAP, DNS.md slice 9, FIRST-PARTY.md gap 23); generated data, large diff, may need a generator script and a pinned Unicode version, to settle in refine. Possible seed impact if the compiler looks names up. Release freeze for 0.1.0 (2026-10-06) holds new PRs, next: refine (queued)[loop:torbscript] refine: still queued behind the intake order. next: waiting (ready buffer full: #4, #163, #164, #223 are ready) until 2026-10-10T04:00:00Z
[loop:torbscript] claim refine by session:loop-torbscript-20261014 until 2026-10-14T00:48Z
orion referenced this issue2026-10-13 22:53:26 +00:00
String.toNfc(),toNfd(),toNfkc()andtoNfkd()(commit 2 of 2) #363text: the full Unicode tablesto text: pinned Unicode data, generator and the character tables (slice 1 of the full Unicode tables)[loop:torbscript] decided by the loop: (1) Unicode 17.0.0 is pinned, not 18.0.0: 17.0.0 has been final for a year and every consumer (UTS #46, NormalizationTest, IdnaTestV2) exists for it; 18.0.0 data files appeared on unicode.org in September 2026 with the release page still marked preliminary. The generator takes the version as one constant, so the bump is one constant plus a regeneration, filed when wanted. (2) The data lives in generated C tables in
runtime/, not in TorbScript tables: a ten-thousand-element literal costs compile time in every program that importsstd/text, and one C implementation keeps the VM and the native back end identical (both read the same natives). (3) Inputs are fetched with pinned SHA-256s and not committed (UnicodeData alone is ~2.2 MB); only the generated file is committed, so review is the generator plus tests. (4) Two-stage table with the block size chosen by measurement (32/64/128/256, smallest wins); budget 48 KiB for this slice, 100 KiB normalization, 60 KiB IDNA data; sizes are measured in forge CI and stated, and a program that does not use a table must not grow by more than 4 KiB (otherwise the tables move into an on-demand object) - runtime objects are all linked today, so this is a real risk and a numeric rule. (5) Semantics:Charkeeps the simple one-to-one mapping (complete now),String.toUpperCase/toLowerCasebecome the full default conversion withFinal_Sigma, no locale tailoring, no parameter;isLetter= category L*,isDigitunchanged. These are the plan the CONCEPT and the doc comments already state ("until milestone 8"), not a break of a promise. (6) Names:toNfc/toNfd/toNfkc/toNfkdas methods ofString(liketoUpperCase, acronyms asUri/Dns); the IDNA lookups are private natives ofstd/dns, and no publicChar.categoryis added in this chain (a 1.0 commitment nobody has asked for; its own follow-up). (7) Size: far too large for one PR, so split into five issues with the dependency order in the body; the natives follow the two-commit rule as separate issues with a published seed in between (precedent #299/#54). No owner question: nothing here is a public statement, cost, external service, product direction or design-record-reserved item.[loop:torbscript] split: normalization is #362 (natives, commit 1 of 2) then #363 (declaration, commit 2 of 2); the IDNA data natives are #364 and the rest of UTS #46 is #365 (depends on #363 and #364); follow-up outside the chain: #366. Only #36 is
ready: the others depend on it, and #363 and #365 also on a published seed.[loop:torbscript] refine: body rewritten to goal, testable criteria, design-record points, seed impact (none for this slice) and size (one PR, two commits); the rest split into #362 to #366 and decided in the comment above, label
readyset, tokens used about 110k (sonnet). next: implementString.toNfc(),toNfd(),toNfkc()andtoNfkd()(commit 2 of 2) #363[loop:torbscript] claim implement by session:loop-torbscript-20261015 until 2026-10-15T04:00Z
[loop:torbscript] implement: PR #368 opened (
Closes #36, two commits,tools/check-commits.shclean), Argus's review requested, forge CI green on the pushed head (tier A, tier B, playground, pdisc; two red rounds fixed on the branch: pdisc stamps of CONCEPT.md/runtime/README.md, the lexer test for a non-ASCII digit, the allocation budget ofcharacter-case).Deviations from the issue, as decided while implementing:
DerivedCoreProperties.txt: Cased and Case_Ignorable are derived properties that Final_Sigma needs.unicode; the driver linksunicode.candunicode_data.conly into a program whose C names one of them (or that hosts the VM), otherwise it definesTORB_WITHOUT_UNICODEand the thunks panic. Nothing underseed/.Size (hello world,
--release): before 369248 B; after, tables not linked 368080 B (-1168); after, tables linked 397136 B (+27888). With lld section GC: 214208 / 212768 / 212840 B.One diagnostic changes: a digit of another script in a name (
count٣) is now an unexpected character instead of the old approximate "letter";compiler/tests/lexer.test.trbis updated.character-case.allocations90 -> 270.Tokens: not measurable from the worker, next: review
[loop:torbscript] close: landed by the merge gate as
c3eee74a31(PR #368); issue closed. #362 and #364 are unblocked. next: done