text: normalization data and natives, teach the compiler (commit 1 of 2 for NFC and NFKC) #362
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Context
Commit 1 of 2 for normalization in
std/text(the declaration is #363).CLAUDE.md"Seed and breaking changes": a new native the std declares has to be known to the seed first. This issue builds the normalization data, the runtime functions and the natives rows, and declares nothing instd/; the declaration waits for a published seed that contains this merge (as #299 / #54 did forBuffer).Part of the chain of #36 (full Unicode tables). Source: FIRST-PARTY.md gap 23 (NFKC for SASLprep), DNS.md section 3 ("NFC normalization"), ROADMAP "the full Unicode tables".
Depends on
#36 (generator, pinned Unicode 17.0.0 data,
tools/fetch-unicode.sh).Decisions (refinement)
torb_text torb_text_to_nfc(torb_text)(also_nfd,_nfkc,_nfkd); nativesString.toNfc,String.toNfd,String.toNfkc,String.toNfkd(naming followstoUpperCase; acronyms are writtenNfclikeUri,Dns,Tls). They are methods ofString, no free functions and noform:parameter.Canonical_Combining_Class, the canonical and compatibility decomposition mappings (UnicodeData.txtfields 3 and 5),Full_Composition_Exclusion(DerivedNormalizationProps.txt); Hangul syllables are computed, not tabled. Generator inputs added totools/unicode/SHA256SUMS.NFC_QC=Yes/ccc=0- the generator adds aquick-checkbit for this.{ccc, flags, decomposition offset}, decomposition strings in oneuint32_tpool, composition as a sorted pair table with binary search. Budget: all normalization data at most 100 KiB; over budget the implementer compacts (pool deduplication, 16-bit pool entries for the BMP) before anything else.Acceptance criteria
runtime/unicode_data.c(or a second generated unit,runtime/unicode_normalization.c, if the size rule of #36 asks for an on-demand object); reproducible as in #36.torb_text_to_nfc/nfd/nfkc/nfkdinruntime/, with ready rows incompiler/src/backend/c/natives.trbforString.toNfcetc. (ownership likeString.toUpperCase);runtime/include/torb_natives.handruntime/machine_natives.cregenerated; the VM calls the same functions.runtime/tests/) runs the wholeNormalizationTest.txtof the pinned version (all four forms, parts 0 to 3, the invariants of the file's header) when the file is present - CI fetches it withtools/fetch-unicode.sh- and a committed excerpt of at least the edge cases (Hangul, ccc reordering, singletons, compatibility, astral) when it is not.compiler/tests/natives.test.trbthat each of the four rows exists, is ready and names its symbol;nativeSymbols()of the VM lists them.nf(nf(x)) == nf(x); NFC of an ASCII text returns the same storage (a runtime test on the pointer).runtime/README.mdand the PR body. The rule of #36 applies: a program that does not use them must not grow by more than 4 KiB.std/,seed/,tests/conformance/. Tier A green; tier B green (runtime touched).seed, or the nightly). Not part of this PR; #363 stays notreadyuntilseeds.txtlists a seed at or after the merge commit.Seed impact
Yes: four new natives. This is commit 1 of the two-commit rule; the std declaration is the next issue.
Size
One PR, two Conventional Commits:
feat(text): normalization data in the generated Unicode tablesandfeat(runtime): NFC, NFD, NFKC and NFKD on texts, and their natives rows.String.toNfc(),toNfd(),toNfkc()andtoNfkd()(commit 2 of 2) #363[loop:torbscript] refine: body written by the refinement of #36 (goal, criteria, decisions, seed impact, size are in it). next: waiting (depends on #36 being merged)
String.toNfc(),toNfd(),toNfkc()andtoNfkd()(commit 2 of 2) #363[loop:torbscript] refine: #36 landed (
c3eee74a31); the body meets the definition of ready, labelreadyset. next: implement[loop:torbscript] claim implement by session:loop-torbscript-20261015 until 2026-10-15T05:52Z
[loop:torbscript] implement: PR #369 (two commits, Closes #362), Argus requested; CI green on
c4e52a6f(tier A/B, gates, whole NormalizationTest.txt: 302 runtime tests, 0 failed); hello-world release +192 bytes (bound 4096), normalization data 86,946 bytes (budget 100 KiB). Tokens: ~1.2M over the whole implement run. next: review[loop:torbscript] close: landed by the merge gate as
e14d7addbf(PR #369); issue closed. #363 stays not ready until a published seed contains this commit. next: doneTlsStream#116