text: public Char.category and the Unicode tables for regex and markdown (follow-up of #36) #366
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Context
Follow-up of #36, not part of its chain.
std/regex(characters.trb, "Unicode as far as the tables reach", TEXT-FORMATS.md) andstd/markdown(text.trb, TEXT-FORMATS.md) keep hand-made tables of general categories, case folding and punctuation "until the Unicode natives of milestone 8". With the generated tables of #36 (landed inc3eee74a) they can read the real data.Goal
A public
Char.category()answering a closedGeneralCategory(the 29 categories a scalar value can have; Unicode's stability policy fixes the set, so the 1.0 rule "a public enum never grows" holds), the simple and full case folding needed by(?i)and link labels, andstd/regex/std/markdownswitched to them with their hand tables deleted.Slices (this issue is the umbrella)
Refined; the order is the dependency order. A slice's PR says
Refs #366, notCloses #366; this issue closes when the last slice has landed.CaseFolding.txtin the generated tables, runtime lookups, four natives rows. Ready. Seed: new natives.std/textdeclarations, commit 2 of 2:GeneralCategory,Char.category(),Char.toCaseFolded(),Char.nextCaseVariant(),String.toCaseFolded(). Waits on #372 merged and in a published seed.std/regex:\p{...}categories and(?i)from the tables. Waits on #373 merged.std/markdown: punctuation, whitespace and label folding from the tables. Waits on #373 merged.#374 and #375 are independent of each other. The API shape (a method
category()againstisPunctuation()-style predicates;toCaseFolded()onCharandStringagainstfoldCase()) is decided in #373, with the reasons.Related
std/dns) add their own tables and natives, including a privategeneralCategory(point)forstd/dns. The slices here and #364 touch the same generator and manifest files but not each other's meaning; the one shared lookup is the category (see #372). Once #373 has landed, #365 may readChar.category()instead of the private native; that is for #365's refinement to decide.\p{Greek}etc. keep their small tables instd/regex), the UnicodeScriptandBidi_Classproperties (#364), normalization (#362, #363).[loop:torbscript] intake: follow-up of #36, not part of its chain, not yet refined. next: waiting (depends on #36 and a published seed; refine when #36 has landed)
[loop:torbscript] claim refine by session:loop-torbscript-20261016 until 2026-10-16T00:03Z
[loop:torbscript] decided by the loop: the shape of the public API, as internal API design under the existing conventions. (1)
Char.category(): GeneralCategoryinstead ofisPunctuation()-style predicates: one lookup answers every question, and the regex classes and CommonMark's "P or S" are written against the seven group predicates of the enum; the enum has the 29 cases a scalar value can have (Cshas none), so it never grows (RELEASE.md public-enum policy). (2) Case folding astoCaseFolded()on bothChar(simple, C+S) andString(full, C+F), mirroring the existingtoUpperCase/toLowerCasepair, notChar.foldCase(); plusChar.nextCaseVariant(), the cyclic successor in a folding class (K, k, Kelvin sign), because(?i)needs the partners of a character and a fold alone has no inverse. (3) No fixedrawValueon the enum, so no number becomes API; the runtime answers anIntand the std maps it. (4) Split into four slices because a new native needs the two-commit rule and a published seed: #372 (compiler and runtime, ready), #373 (std/text, waits on a published seed), #374 (regex) and #375 (markdown), both waiting on #373. Numbers: tables 26 456 bytes today, folding data budget at most +16 KiB, a program using none of the natives at most +4 KiB; regex\p{L}compile budget 1 s in the VM / 50 ms native, set in #374.[loop:torbscript] refine: umbrella body rewritten, split into #372 (teach the compiler, commit 1 of 2, ready), #373 (std/text declarations, commit 2 of 2, waits on a published seed), #374 (std/regex) and #375 (std/markdown), both waiting on #373 merged; #366 closes when the last slice has landed. next: waiting (slices #372 to #375 carry the work). Tokens: ~150k