text: Char.category(), GeneralCategory and case folding in std/text (commit 2 of 2 for #366) #373
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Context
Commit 2 of 2 for #366.
std/textgets the public general category of aCharand the case folding that the natives of #372 answer, with the docs and conformance programs that go with them.std/regexandstd/markdownswitch to them in two later issues. The text of #366 left the shape of the API to refinement: it is internal API design under the existing naming conventions (thetoUpperCase/toLowerCasepair exists onCharand onString), decided below.Depends on
#372 (the natives rows), merged and in a published seed (
seeds.txton theseedsrelease lists a seed at or after the merge commit). Until then this issue is notready.Decisions (refinement)
Char.category(): GeneralCategory, a closed public enum instd/textwith the 29 categories a scalar value can have;Cs(surrogate) has no case because aCharis never a surrogate. A public enum never grows within 1.x (RELEASE.md section 3 and the policy row); the Unicode Stability Policy fixes the set of values, so the policy holds. It is a method onChar(methods are the preferred API style, #146), not a set ofisPunctuation()-style predicates onChar: one lookup answers every question, andChar.isLetter()stays as the fast path that exists.stdname theirs:UppercaseLetter,LowercaseLetter,TitlecaseLetter,ModifierLetter,OtherLetter,NonspacingMark,SpacingMark,EnclosingMark,DecimalNumber,LetterNumber,OtherNumber,ConnectorPunctuation,DashPunctuation,OpenPunctuation,ClosePunctuation,InitialPunctuation,FinalPunctuation,OtherPunctuation,MathSymbol,CurrencySymbol,ModifierSymbol,OtherSymbol,SpaceSeparator,LineSeparator,ParagraphSeparator,Control,Format,PrivateUse,Unassigned.with Equals, Hash, Show.rawValue()would publish a number nobody should rely on.Char.category()is ordinary TorbScript over a private nativeChar.categoryCode(): Int(amatchover the runtime's numbers,Cs= 27 unreachable). If the manifest cannot name a private native ofChar, the implementer makes it a documentednative fnof the same name and reports it in the PR.GeneralCategory:abbreviation(): String("Lu"),static fn fromAbbreviation(text: String): Self?(Nonefor a name that is no category, and for"Cs"), and the seven groups of the standard as predicates:isLetter(),isMark(),isNumber(),isPunctuation(),isSymbol(),isSeparator(),isOther(). Regex classes (\p{L}) and CommonMark ("P or S") are written against exactly these.Char.toCaseFolded(): Self(simple folding, one code point in and out) andString.toCaseFolded(): Self(full folding,ßbecomesss); notChar.foldCase(). The name says what comes out; caseless comparison isa.toCaseFolded() == b.toCaseFolded(), which is nottoLowerCase()(ςandσ, Cherokee,ẞ).Char.nextCaseVariant(): Self, the cyclic successor in the class of code points that fold alike (K->k-> Kelvin sign ->K); it is public becausestd/regexis a separate package and a private native cannot be reached from it. Doc comment names it as the primitive for case-insensitive matching.caseVariants()list and noisPunctuation()onChar: additions are cheap, a public commitment is not, and the callers (the next two issues) need neither.Acceptance criteria
std/text/src/lib.trb:GeneralCategorywith the members above;Char.category(), privatenative fn categoryCode(),native fn toCaseFolded()andnative fn nextCaseVariant()onChar,native fn toCaseFolded()onString, each with a doc comment in the style of the neighbours (# Examples;# Pitfalls: simple against full folding, folding is not lower-casing, no locale/Turkish tailoring, normalization is a separate question).GeneralCategorycase is reached and maps to the right abbreviation: a conformance programtests/conformance/character-category.trb(+.expected,.allocations), identical in the VM and natively, with one code point per category ('A','a',DžU+01C5,ʰU+02B0,אU+05D0, U+0301, U+0903, U+20DD,٣U+0663,ⅧU+2167,½,_,-,(,),«,»,!,+,$,^,©, space, U+2028, U+2029, U+0000, U+00AD, U+E000, U+0378),fromAbbreviationof every abbreviation and of"Cs","xx"and the empty text, the group predicates, and a sweep of the Basic Multilingual Plane thatisLetter()agrees withcategory().isLetter()on every code point.tests/conformance/character-case-folding.trb(+ files), VM and native identical:K/k/Kelvin sign cycle throughnextCaseVariant,Σ/σ/ς,ßandẞ(Charanswers itself or its simple folding,Stringanswersss),ffi,İ(no Turkic folding), Armenian, Cherokee, Georgian, ASCII text (storage shared, no allocation), the empty text, bytes that are no character, idempotence on a mixed text.docs/standard-library/text.md, the status lines ofdocs/ROADMAP.mdandCONCEPT.md("Case mapping and character classification read the Unicode Character Database") say what is built;build/release/torb docs skill docs skillsregenerated and committed (--checkis a tier A gate).Seed impact
No new native here (the rows of #372 are known to the seed once it is published), but a std name is added: the declarations are what commit 1 made safe. Nothing under
seed/; no seed refresh needed after this merge. The compiler looks up none of the new names by string.Size
One PR, one Conventional Commit:
feat(text): Char.category and case folding.Part of #366.
[loop:torbscript] refine: body written by the refinement of #366 (API shape decided there:
Char.category()with a closedGeneralCategoryof 29 cases,toCaseFolded()onCharandString,nextCaseVariant(); goal, criteria, seed impact, size). next: waiting (depends on #372 merged and a published seed containing it)