feat(text): Unicode case folding data and the natives for Char.category and case folding #378
argus
Loading…
Reference in a new issue
No description provided.
Delete branch "loop/372-unicode-case-folding"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #372
Commit 1 of 2 for #366: the generated Unicode tables get case folding, and the compiler learns the four natives that read it. The declarations in
std/follow in #373; nothing understd/,seed/ortests/conformance/changes.What changes
tools/unicode/generate.trbreads CaseFolding.txt (17.0.0), pinned intools/unicode/SHA256SUMS(fetch-unicode.shreads the hash from there) and listed with its hash in the header of the generated files. The output is byte for byte reproducible.{upper, lower, fold, next, category, flags}:foldis the simple folding (statuses C and S),nextthe delta to the next member of the fold class in code point order (cyclic; 0 when alone), and a new flag marks the 104 code points of status F, which are a sorted table of their own. The generator checks that both foldings are idempotent and fails above a budget of 43 008 bytes.runtime/unicode.c:torb_char_category_code,torb_char_to_case_folded,torb_char_next_case_variantandtorb_text_to_case_folded(full folding, statuses C and F; bytes that are no character are copied; status T is not read).compiler/src/backend/c/natives.trb: four ready rows (Char.categoryCode,Char.toCaseFolded,Char.nextCaseVariant,String.toCaseFolded), added tounicodeSymbols;torb_natives.handmachine_natives.cregenerated withtorb natives; the VM thunks stay guarded byTORB_WITHOUT_UNICODE.runtime/tests/unicode_test.ccompares the category and the simple and full folding of every code point with the pinned UnicodeData.txt and CaseFolding.txt, or with the committed excerpts (unicode-excerpt.txt, newcasefolding-excerpt.txtwith statuses C, S, F and T). It also checks that folding is idempotent, that every class closes undernextCaseVariant, that two code points share a class exactly when their simple foldings are equal, and that the largest class has 4 members (the thetas).compiler/tests/natives.test.trbcovers the four rows.runtime/README.md("The Unicode tables") anddocs/BACKEND.mddescribe the inputs, sizes and natives.#364 (IDNA)
IDNA landed first and keeps its own category column in
idna_data.c, indexed by IDNA status.Char.categoryCodereads the general category of the Unicode tables; the data is not duplicated into or out of the IDNA tables, which read more than the category.Size
Tables: 26 456 bytes before, 30 676 after (+4 220 bytes; budget +16 KiB for the folding and 42 KiB in all). The numbers of the program that uses none of the natives and of the objects come from
tools/size-cost.shin CI tier B, below.abc4e2471a1ac363d76fApproved at
1ac363d.unicode_test.cchecks category, simple and full folding of every code point against the pinned files (or the committed excerpts), idempotence of text folding, class closure undernextCaseVariant, the class-equals-equal-folding property and the largest class;natives.test.trbcovers the four rows. CI green for this head (tier A/B, bootstrap, pdisc, gates).std/,seed/ortests/conformance/.SHA256SUMS; no secrets; size budget enforced in the generator (+4 220 bytes).Non-blocking:
addFoldingschecks only the simple one; the full folding is covered by the C test (twice-folded text equals once-folded text). Either word it that way or add the check to the generator.runtime/tests/unicode_check.h: a few lines now exceed the surrounding width.