text: Unicode case folding data and the natives for Char.category and case folding (teach the compiler, commit 1 of 2 for #366) #372
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Context
Commit 1 of 2 for the public
Char.category()and the case folding of #366 (the declarations instd/textare #373;std/regexandstd/markdownswitch in #374 and #375). #36 generated the Unicode tables and madeChar.isLetter,isWhitespace,toUpperCase,toLowerCaseand the twoStringconversions read them (runtime/README.md, "The Unicode tables"). The general category of every code point is already in those tables (torb_unicode_record.category, 30 values,runtime/include/torb_unicode_data.h); the case folding is not. This issue adds the folding data and the runtime lookups, and the natives rows for four lookups. It declares nothing instd/: a new native is commit 1 of 2 as everywhere (CLAUDE.md, "Seed and breaking changes").Depends on
#36 (landed in
c3eee74a, generator and tables). Needs nothing unpublished.Decisions (refinement)
Int/Char/Stringin and out, no list or enum built by the runtime (a runtime function cannot build a type of the program, BACKEND 3.7):Char.categoryCode(): Int(private in the later declaration; the number of the general category as the header numbers them,Lu= 0 toCn= 29, a value above U+10FFFF isCn),Char.toCaseFolded(): Char(simple case folding,CaseFolding.txtstatuses C and S),Char.nextCaseVariant(): Char(the next code point of the same simple-folding class, see below) andString.toCaseFolded(): String(full case folding, statuses C and F).Stringhas no locale, as fortoUpperCase.nextCaseVariantis the cyclic successor in code point order inside one class of code points that fold to the same code point (the idea of Go'sunicode.SimpleFold, which RE2's(?i)is built on):K(U+004B) ->k(U+006B) ->KKelvin sign (U+212A) ->K. A code point with no partner answers itself. This is what letsstd/regexbuild(?i)without scanning code points, and it needs no inverse table of the folding.Charnatives if #364 has landed first: #364 adds ageneralCategory(point)lookup forstd/dns. Whichever lands first owns the C function; the second points its natives row at the same function and does not copy the lookup. The tables stay one unit (unicode.c,unicode_data.c), linked only where the emitted C names one ofunicodeSymbols.CaseFolding.txtof 17.0.0, with its SHA-256 intools/unicode/SHA256SUMSandtools/fetch-unicode.sh.Acceptance criteria
tools/unicode/generate.trbreadsCaseFolding.txtand writes the folding data intoruntime/unicode_data.cand its header, byte for byte reproducible as in #36; the header lists the new input and its hash.runtime/unicode.chas the four lookups;torb_char_category_codeanswers the number of the header's enumeration;Char.toCaseFoldedandString.toCaseFoldedare idempotent (folding a folded text changes nothing);String.toCaseFoldedcopies bytes that are no character astoUpperCasedoes and may answer a longer text (ßandẞfold toss,ffitoffi).compiler/src/backend/c/natives.trb(Char.categoryCode,Char.toCaseFolded,Char.nextCaseVariant,String.toCaseFolded), the four C names added tounicodeSymbolsso the tables link on demand, the VM thunks for a buildTORB_WITHOUT_UNICODE,torb_natives.handmachine_natives.cregenerated; VM and native agree.runtime/tests/unicode_test.c) that compares category, simple and full folding of every code point with the pinnedUnicodeData.txtandCaseFolding.txtwhen present (CI fetches them) and with a committed excerpt otherwise. The excerpt has all four statuses C, S, F and T (T must not change the answer), the cycleK/k/K,Σ/σ/ς,θ/ϑ/ϴ/Θ,Dž,ß,ẞ,ffi,İ, an Armenian pair, a Cherokee pair, a Georgian pair (Mtavruli to Mkhedruli), and one code point of each of the 29 categories aCharcan have (all butCs). The test also checks that every orbit closes (followingnextCaseVariantreturns to the start) and that two code points share an orbit exactly when their simple folding is equal, and states the largest orbit it finds.compiler/tests/natives.test.trb: each of the four rows exists and is ready;runtime/tests/natives_header_test.ccompiles.runtime/README.md("The Unicode tables": inputs, sizes, the natives) anddocs/BACKEND.mdwhere it lists the natives are updated.std/,seed/ortests/conformance/. Tier A and B green. After the merge the seed is published (forge Actions ->seed); that is not part of this PR and is what #373 waits for.Seed impact
Yes: new natives (commit 1 of 2). The
std/textdeclarations must not land before a seed at or after this merge is published.Collision with #364
#364 (IDNA data) extends the same generator,
torb_natives.h,machine_natives.c,runtime/README.mdand the manifest. They are independent in meaning and either may land first; the one that lands second rebases (linear history). The only shared lookup is the category, see the decision above.Size
One PR, two Conventional Commits:
feat(text): case folding data in the generated Unicode tablesandfeat(runtime): category and case folding lookups and their natives rows.Part of #366.
[loop:torbscript] decided by the loop: the only dependency, #36, has landed (
c3eee74a), the pinned-data generator and the table unit exist, and the issue needs nothing unpublished; the new natives are commit 1 of 2 under the two-commit rule and the seed refresh stays with the owner. Goal, criteria, decisions (four natives,nextCaseVariantas the cyclic successor, table budget at most +16 KiB over today's 26 456 bytes, at most 4 KiB for a program that uses none), seed impact and size are in the body. Definition of ready met, labelledready. Part of #366.[loop:torbscript] claim implement by session:loop-torbscript-20261017 until 2026-10-17T04:10:30Z
[loop:torbscript] implement: PR #378 (#378), two commits, CI green at
1ac363d7, approved and merged by argus; data and generator (CaseFolding.txt pinned in SHA256SUMS), the four lookups in unicode.c, four ready rows in natives.trb, regenerated torb_natives.h/machine_natives.c, tests and README/BACKEND updated; #364's category column was not duplicated. Measured in CI tier B: tables 26 456 -> 30 676 bytes (+4 220, bound +16 KiB), unicode_data.o 26 528 -> 30 752, unicode.o code 2 598 -> 3 257; hello-world (none of the natives) 376 792 -> 381 080 bytes = +4 288 against a bound of 4 096, which the tier B line reports. The growth is not the unicode part (hello's C names none of the ten symbols, so unicode.c/unicode_data.c are not linked) but the always-linked machine_natives.c, which gained four table entries and four stubs and renumbered its thunks, plus page alignment of the binary (4 288 = 4 096 + 192). Not mitigated in this PR; a possible follow-up is one shared stub for the TORB_WITHOUT_UNICODE thunks in the generator. next: waiting (decision on the 192-byte overshoot of the hello-world bound, token use of this worker about 1.3M)