markdown: Unicode punctuation, whitespace and label case folding from the tables (slice of #366) #375

New issue

Open

opened 2026-10-15 22:08:32 +00:00 by orion · 1 comment

orion commented

2026-10-15 22:08:32 +00:00

Collaborator

Context

Part of #366 (declarations: #373). std/markdown/src/text.trb keeps TorbScript tables for what CommonMark asks of Unicode: isPunctuation (the categories P and S) over a list of blocks, isSymbolOfLetterlike, isUnicodeWhitespace, and the case folding of link labels (folded) over ASCII-and-Greek-and-Cyrillic (TEXT-FORMATS.md). The spec defines all three from the Unicode data, so the tables are replaced by it.

Depends on

The std/text declarations of #373 (Char.category(), GeneralCategory, String.toCaseFolded()), merged. Independent of #374.

Decisions (refinement)

Unicode punctuation is category().isPunctuation() || category().isSymbol() (CommonMark: the categories P and S), with the existing ASCII fast path kept. Unicode whitespace is SpaceSeparator or tab, line feed, form feed, carriage return.
A link label is whitespace-collapsed, then String.toCaseFolded() (the full folding the spec's "Unicode case fold" means: ẞ and ß fold to ss), not toLowerCase(). The order of the steps stays as it is today.
isAsciiPunctuation, isAsciiLetter, isSpaceOrTab are not Unicode tables and stay.

Acceptance criteria

isPunctuation, isUnicodeWhitespace and normalizedLabel use the natives above; isSymbolOfLetterlike, folded and every block list of std/markdown/src/text.trb are deleted.
Every CommonMark spec example that passes today still passes (std/markdown/tests/spec.test.trb; the PR states the count from CI), and the writer's tests are unchanged.
New tests (VM and native) for what the old tables did not reach: emphasis flanking next to a punctuation or symbol outside the old blocks (U+A4FE, a Lisu punctuation mark; a currency sign such as ₿ U+20BF), a label that matches across case in Armenian, Cherokee and Georgian ([ԱԲ] against a definition [աբ]), [ẞ] against [SS], a no-break space and an ideographic space as Unicode whitespace.
docs/design/TEXT-FORMATS.md (the bullet "Unicode punctuation ... are TorbScript tables") and the markdown page of docs/standard-library/ say the data is the Unicode tables; skills regenerated.
Tier A and B green in forge CI.

Seed impact

None: std/markdown only.

Size

One PR, one Conventional Commit: feat(markdown): Unicode punctuation, whitespace and label folding from the tables.

## Context Part of #366 (declarations: #373). `std/markdown/src/text.trb` keeps TorbScript tables for what CommonMark asks of Unicode: `isPunctuation` (the categories P and S) over a list of blocks, `isSymbolOfLetterlike`, `isUnicodeWhitespace`, and the case folding of link labels (`folded`) over ASCII-and-Greek-and-Cyrillic ([TEXT-FORMATS.md](https://git.torb.dev/torbscript/language/src/commit/740a15aad627ea4ff37f857a5a68d780c2b56aed/docs/design/TEXT-FORMATS.md#L150)). The spec defines all three from the Unicode data, so the tables are replaced by it. ## Depends on The `std/text` declarations of #373 (`Char.category()`, `GeneralCategory`, `String.toCaseFolded()`), **merged**. Independent of #374. ## Decisions (refinement) - **Unicode punctuation is `category().isPunctuation() || category().isSymbol()`** (CommonMark: the categories P and S), with the existing ASCII fast path kept. **Unicode whitespace is `SpaceSeparator` or tab, line feed, form feed, carriage return.** - **A link label is whitespace-collapsed, then `String.toCaseFolded()`** (the full folding the spec's "Unicode case fold" means: `ẞ` and `ß` fold to `ss`), not `toLowerCase()`. The order of the steps stays as it is today. - `isAsciiPunctuation`, `isAsciiLetter`, `isSpaceOrTab` are not Unicode tables and stay. ## Acceptance criteria - [ ] `isPunctuation`, `isUnicodeWhitespace` and `normalizedLabel` use the natives above; `isSymbolOfLetterlike`, `folded` and every block list of `std/markdown/src/text.trb` are deleted. - [ ] Every CommonMark spec example that passes today still passes (`std/markdown/tests/spec.test.trb`; the PR states the count from CI), and the writer's tests are unchanged. - [ ] New tests (VM and native) for what the old tables did not reach: emphasis flanking next to a punctuation or symbol outside the old blocks (U+A4FE, a Lisu punctuation mark; a currency sign such as `₿` U+20BF), a label that matches across case in Armenian, Cherokee and Georgian (`[ԱԲ]` against a definition `[աբ]`), `[ẞ]` against `[SS]`, a no-break space and an ideographic space as Unicode whitespace. - [ ] `docs/design/TEXT-FORMATS.md` (the bullet "Unicode punctuation ... are TorbScript tables") and the markdown page of `docs/standard-library/` say the data is the Unicode tables; skills regenerated. - [ ] Tier A and B green in forge CI. ## Seed impact None: `std/markdown` only. ## Size One PR, one Conventional Commit: `feat(markdown): Unicode punctuation, whitespace and label folding from the tables`.

orion added the

loop

label

2026-10-15 22:08:32 +00:00

orion referenced this issue

2026-10-15 22:08:44 +00:00

text: Unicode case folding data and the natives for Char.category and case folding (teach the compiler, commit 1 of 2 for #366) #372

orion referenced this issue

2026-10-15 22:08:44 +00:00

text: public Char.category and the Unicode tables for regex and markdown (follow-up of #36) #366

orion commented

2026-10-15 22:08:56 +00:00

Author

Collaborator

[loop:torbscript] refine: body written by the refinement of #366 (decisions: P or S via the category, labels folded with String.toCaseFolded()). next: waiting (depends on #373 merged)

[loop:torbscript] refine: body written by the refinement of #366 (decisions: P or S via the category, labels folded with `String.toCaseFolded()`). next: waiting (depends on #373 merged)

orion referenced this issue

2026-10-15 22:08:57 +00:00

text: public Char.category and the Unicode tables for regex and markdown (follow-up of #36) #366

orion referenced this issue

2026-10-15 22:08:57 +00:00

text: public Char.category and the Unicode tables for regex and markdown (follow-up of #36) #366