regex: Unicode categories for \p{...} and case folding from the tables (slice of #366) #374
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Context
Part of #366 (declarations: #373).
std/regexkeeps hand tables instd/regex/src/characters.trb:\p{...}knowsAny,L,Lu,Ll,N,Nd,Z,Zs,Latin,GreekandCyrillic, and(?i)folds byfoldOrbit, a hand list over ASCII, Latin-1, Latin Extended-A, Greek and Cyrillic (TEXT-FORMATS.md, "Unicode as far as the tables reach"). WithChar.category()andChar.nextCaseVariant()it reads the real data.Depends on
The
std/textdeclarations of #373 (Char.category(),GeneralCategory,Char.nextCaseVariant()), merged. A published seed is not needed beyond what that issue needed: this change is std only.Decisions (refinement)
\p{X}takes the Unicode abbreviations (Lu,Nd,Zs, ...) and the seven groupsL,M,N,P,S,Z,C, with\P{X}and the negations the parser accepts today; no long names, as in RE2. Names and groups follow RE2:CisCc,Cf,CoandCs, and\p{Cs}is accepted and matches nothing because aCharis never a surrogate.Cn(unassigned) is an accepted extension that RE2 lacks and is not part ofC, so a pattern ported from RE2 means the same.\d,\w,\s,\band the POSIX classes stay ASCII, as in RE2;Anystays.Latin,GreekandCyrillickeep their hand tables: script data is not in the pinned input set of the tables (Scripts.txtarrives with #364 for five scripts only). They are the only hand tables left, and the docs say so.(?i)is the simple case folding orbit of every code point,Char.nextCaseVariant()followed until it returns to the start, and no longer bounded by afoldLimit.(?i)\p{Lu}matchesa, as in RE2.Char.category()(about 1.1 million lookups per class, probably too slow in the VM), and (b) a category mask onCharacterSetchecked at match time, with(?i)as a flag that tests the orbit of the input character at match time instead of adding the partners of every range at compile time. The recommendation is (b); the implementer measures both in forge CI and states the numbers. Budget set here: compiling a pattern with\p{L}takes at most 1 s in the VM and at most 50 ms natively, and no regex test grows by more than 10% of its time because of the tables.Acceptance criteria
unicodeSetanswers every abbreviation and group above throughChar.category();letterSet,upperSet,lowerSet,decimalSet,numberSet, theZset andfoldOrbitwith its hand data are deleted fromstd/regex/src/characters.trb.\p{Lu},\p{Ll},\p{Nd}on Arabic-Indic and fullwidth digits,\p{Pd},\p{Sc}on$,€and₿,\p{M}on a combining mark,\p{L}on Hebrew, CJK and Hangul text,\P{L},[^\p{L}\d],\p{Cs}matching nothing,\p{Cn}on U+0378, an unknown name still refused with the same error;(?i)on Armenian, Cherokee, Georgian,Dž,Σσς, the Kelvin sign, and(?i)[a-z]matchingKKelvin sign as today.std/regex/tests/search-cases.trb) still pass in full (1808 of 1808, asTEXT-FORMATS.mdstates); the other regex tests pass unchanged.docs/design/TEXT-FORMATS.md("Unicode as far as the tables reach") anddocs/standard-library/regex page say which names and folding are supported; the skills are regenerated (torb docs skill docs skills).Seed impact
None:
std/regexonly, no native, no name the compiler looks up by string.Size
One PR, one Conventional Commit:
feat(regex): Unicode categories and case folding from the tables. If the mask representation growsCharacterSetbeyond a reviewable diff, two commits:refactor(regex): sets with a category maskand the switch.[loop:torbscript] refine: body written by the refinement of #366 (decisions: RE2 names and groups, scripts keep their tables,
(?i)by case orbit, representation chosen by measurement with a stated budget). next: waiting (depends on #373 merged)