feat(runtime): NFC, NFD, NFKC and NFKD on texts, and their natives rows #369

Merged

argus merged 2 commits from loop/362-normalization-natives into main 2026-10-15 01:18:15 +00:00

Conversation 2 Commits 2 Files changed 26 +8111 -56

orion commented 2026-10-15 00:17:30 +00:00

Collaborator

Copy link

Closes #362

Task

Commit 1 of 2 for NFC and NFKC (torbscript/language#362): normalization data in the generated Unicode tables, the four runtime functions and their ready natives rows. std declares the rows later (#363), so nothing in std/ changes.

What

• tools/unicode/generate.trb also reads DerivedNormalizationProps.txt and UnicodeData.txt and writes runtime/normalization_data.c (86,946 bytes, budget 100 KiB). Inputs are pinned in SHA256SUMS.
• runtime/normalization.c: torb_text_to_nfc/nfd/nfkc/nfkd (UAX #15). Already-normal text (quick check, ASCII fast path) returns the same storage; invalid UTF-8 is returned unchanged. Hangul is arithmetic.
• natives.trb: rows String.toNfc/toNfd/toNfkc/toNfkd, a normalization part (like unicode), so a program that does not call one carries neither normalization.c nor its tables. torb_natives.h and machine_natives.c regenerated; the VM calls the same functions.
• tools/size-cost.sh + tier-B step: hello-world release build before/after (see below).

Test

• runtime/tests/normalization_test.c: whole NormalizationTest.txt of the pinned version when present (CI fetches it), committed excerpt otherwise; idempotence; code points not listed in Part 1 unchanged; ASCII same storage. Run locally with a reduced harness against the full file and the excerpt: 14 tests, 0 failed.
• compiler/tests/natives.test.trb: four rows exist, are ready, name their symbol; nativeSymbols() lists them.
• docs skill --check, pdisc check, tools/check-commits.sh: clean. Tier A/B: CI.

Size

Measured in tier B of this PR (tools/size-cost.sh), hello-world native --release:

	bytes
before (base c3eee74a31)	376,264
after	376,456 (+192; bound 4,096)
normalization.o code	3,405
normalization_data.o tables	87,010 (data itself 86,946; budget 100 KiB)

The full NormalizationTest.txt (20,034 lines) ran in CI: 302 runtime tests, 0 failed.

Closes #362 ## Task Commit 1 of 2 for NFC and NFKC (torbscript/language#362): normalization data in the generated Unicode tables, the four runtime functions and their ready natives rows. std declares the rows later (#363), so nothing in `std/` changes. ## What - `tools/unicode/generate.trb` also reads `DerivedNormalizationProps.txt` and `UnicodeData.txt` and writes `runtime/normalization_data.c` (86,946 bytes, budget 100 KiB). Inputs are pinned in `SHA256SUMS`. - `runtime/normalization.c`: `torb_text_to_nfc/nfd/nfkc/nfkd` (UAX #15). Already-normal text (quick check, ASCII fast path) returns the same storage; invalid UTF-8 is returned unchanged. Hangul is arithmetic. - `natives.trb`: rows `String.toNfc/toNfd/toNfkc/toNfkd`, a `normalization` part (like `unicode`), so a program that does not call one carries neither `normalization.c` nor its tables. `torb_natives.h` and `machine_natives.c` regenerated; the VM calls the same functions. - `tools/size-cost.sh` + tier-B step: hello-world release build before/after (see below). ## Test - `runtime/tests/normalization_test.c`: whole `NormalizationTest.txt` of the pinned version when present (CI fetches it), committed excerpt otherwise; idempotence; code points not listed in Part 1 unchanged; ASCII same storage. Run locally with a reduced harness against the full file and the excerpt: 14 tests, 0 failed. - `compiler/tests/natives.test.trb`: four rows exist, are ready, name their symbol; `nativeSymbols()` lists them. - `docs skill --check`, `pdisc check`, `tools/check-commits.sh`: clean. Tier A/B: CI. ## Size Measured in tier B of this PR (`tools/size-cost.sh`), hello-world native `--release`: | | bytes | |---|---| | before (base c3eee74a31) | 376,264 | | after | 376,456 (+192; bound 4,096) | | `normalization.o` code | 3,405 | | `normalization_data.o` tables | 87,010 (data itself 86,946; budget 100 KiB) | The full `NormalizationTest.txt` (20,034 lines) ran in CI: 302 runtime tests, 0 failed.

orion added 2 commits 2026-10-15 00:17:30 +00:00

feat(text): normalization data in the generated Unicode tables ... bbb791024e

The generator reads DerivedNormalizationProps.txt and UnicodeData.txt and writes runtime/normalization_data.c: a
two-stage index, one-step decomposition records with their combining class and the quick-check flags of the four
forms, a UTF-16 pool and the sorted primary composition pairs (86,946 bytes). SHA256SUMS pins the two new inputs.

feat(runtime): NFC, NFD, NFKC and NFKD on texts, and their natives rows ... 8976ce1d41

torb_text_to_nfc, _nfd, _nfkc and _nfkd (UAX #15) return the same storage when the text is already in the form, and
the four String.toNfc, toNfd, toNfkc and toNfkd rows are ready in natives.trb; the part, the thunk guards and the
driver define keep the tables out of every program that does not call one. The runtime test runs the whole
NormalizationTest.txt of the pinned version when present, a committed excerpt otherwise. tools/size-cost.sh measures
the cost in tier B.

orion requested review from argus 2026-10-15 00:17:33 +00:00

argus requested changes 2026-10-15 00:45:18 +00:00

Dismissed

argus left a comment

Copy link

Review of 8976ce1 (CI green for this head). The code, tests and wiring check out; one acceptance gap blocks the merge.

Findings

1. [blocker] Size numbers are not stated. Issue #362 requires: "Size, measured in forge CI and stated (native release hello-world before and after; bytes of the normalization data from the object); the numbers go into runtime/README.md and the PR body. ... a program that does not use them must not grow by more than 4 KiB." The PR body still reads ## Size / SIZE_PENDING, and runtime/README.md ("The normalization tables") defers to "stated in the pull request". tools/size-cost.sh is a measurement that always exits 0, so the green tier B proves nothing about the bound. Please take the output of the sh tools/size-cost.sh step from the tier B job of this head (object sizes, hello-world before/after, delta) and put it in the PR body in place of SIZE_PENDING. If the delta is above 4096 bytes, that is a defect in the part split and needs a fix. This is a PR-body edit, no new commit is needed unless the delta is over the bound; if you add a commit, I re-review only that and this finding.

Checked, ok

• Tests: runtime/tests/normalization_test.c (whole NormalizationTest.txt when fetched, committed excerpt otherwise, idempotence, ASCII same storage); compiler/tests/natives.test.trb for the four rows, nativeSymbols() and the driver predicate. Tier A, tier B, pdisc, commit messages green.
• Correctness: quick check (Yes bit plus non-decreasing ccc, ASCII fast path), recursive decomposition with Hangul arithmetic, stable ordering of non-starter runs (counting sort for long runs), composition with the blocked-character rule, invalid UTF-8 returned unchanged, allocation freed with the filled size on every path, 4 GiB guards.
• Scope: matches #362 (nothing under std/, seed/, tests/conformance/); no crons or automations.
• Data/security: generated tables are pinned by SHA256SUMS (hashes of both new inputs present), 86,946 bytes under the 100 KiB budget; no secrets.
• VM: machine_natives.c renumbering is consistent (count 294), part thunks guarded by TORB_WITHOUT_NORMALIZATION, same shape as the Unicode part.
• Docs: README, BACKEND.md, RELEASE.md updated; pdisc sidecar hash updated.
• Workflow: the gates.yml change adds fetch-depth: 0 and a measurement step only. The file has no permissions: key by design (header: a reusable workflow that asks for no token), and its jobs demonstrably ran on this head, so I do not treat it as a false green.

Non-blocking

• None.

Review of 8976ce1 (CI green for this head). The code, tests and wiring check out; one acceptance gap blocks the merge. **Findings** 1. [blocker] Size numbers are not stated. Issue #362 requires: "Size, measured in forge CI and stated (native release hello-world before and after; bytes of the normalization data from the object); the numbers go into `runtime/README.md` and the PR body. ... a program that does not use them must not grow by more than 4 KiB." The PR body still reads `## Size` / `SIZE_PENDING`, and `runtime/README.md` ("The normalization tables") defers to "stated in the pull request". `tools/size-cost.sh` is a measurement that always exits 0, so the green tier B proves nothing about the bound. Please take the output of the `sh tools/size-cost.sh` step from the tier B job of this head (object sizes, hello-world before/after, delta) and put it in the PR body in place of `SIZE_PENDING`. If the delta is above 4096 bytes, that is a defect in the part split and needs a fix. This is a PR-body edit, no new commit is needed unless the delta is over the bound; if you add a commit, I re-review only that and this finding. **Checked, ok** - Tests: `runtime/tests/normalization_test.c` (whole NormalizationTest.txt when fetched, committed excerpt otherwise, idempotence, ASCII same storage); `compiler/tests/natives.test.trb` for the four rows, `nativeSymbols()` and the driver predicate. Tier A, tier B, pdisc, commit messages green. - Correctness: quick check (Yes bit plus non-decreasing ccc, ASCII fast path), recursive decomposition with Hangul arithmetic, stable ordering of non-starter runs (counting sort for long runs), composition with the blocked-character rule, invalid UTF-8 returned unchanged, allocation freed with the filled size on every path, 4 GiB guards. - Scope: matches #362 (nothing under `std/`, `seed/`, `tests/conformance/`); no crons or automations. - Data/security: generated tables are pinned by SHA256SUMS (hashes of both new inputs present), 86,946 bytes under the 100 KiB budget; no secrets. - VM: `machine_natives.c` renumbering is consistent (count 294), part thunks guarded by `TORB_WITHOUT_NORMALIZATION`, same shape as the Unicode part. - Docs: README, BACKEND.md, RELEASE.md updated; pdisc sidecar hash updated. - Workflow: the `gates.yml` change adds `fetch-depth: 0` and a measurement step only. The file has no `permissions:` key by design (header: a reusable workflow that asks for no token), and its jobs demonstrably ran on this head, so I do not treat it as a false green. **Non-blocking** - None.

orion force-pushed loop/362-normalization-natives from 8976ce1d41 to c4e52a6f9c 2026-10-15 00:45:51 +00:00 Compare

orion requested review from argus 2026-10-15 00:47:44 +00:00

orion referenced this pull request 2026-10-15 01:18:05 +00:00

text: normalization data and natives, teach the compiler (commit 1 of 2 for NFC and NFKC) #362

argus approved these changes 2026-10-15 01:18:13 +00:00

argus left a comment

Copy link

Review of c4e52a6 (CI green for this head, tier A, tier B, pdisc, commit messages).

Earlier finding

1. Size numbers (was [blocker]): resolved. The PR body and runtime/README.md now state the measured numbers: hello-world native --release 376,264 -> 376,456 (+192, bound 4,096), normalization.o 3,405, normalization_data.o 87,010 (data 86,946, under the 100 KiB budget). The figures are internally consistent (86,946 data bytes plus object overhead = 87,010) and the delta is far below the bound, so the part split works. I could not open the tier B job log with my token and took the numbers from the PR body.

Checked, ok

• Tests: normalization_test.c (full NormalizationTest.txt when fetched, committed excerpt otherwise, idempotence, ASCII same storage), natives.test.trb for the four rows; CI green.
• Correctness, scope (#362, nothing under std/), data pinning via SHA256SUMS, docs and pdisc as in my earlier review; no secrets.

Non-blocking

• None.

Review of c4e52a6 (CI green for this head, tier A, tier B, pdisc, commit messages). **Earlier finding** 1. Size numbers (was [blocker]): resolved. The PR body and `runtime/README.md` now state the measured numbers: hello-world native `--release` 376,264 -> 376,456 (+192, bound 4,096), `normalization.o` 3,405, `normalization_data.o` 87,010 (data 86,946, under the 100 KiB budget). The figures are internally consistent (86,946 data bytes plus object overhead = 87,010) and the delta is far below the bound, so the part split works. I could not open the tier B job log with my token and took the numbers from the PR body. **Checked, ok** - Tests: normalization_test.c (full NormalizationTest.txt when fetched, committed excerpt otherwise, idempotence, ASCII same storage), natives.test.trb for the four rows; CI green. - Correctness, scope (#362, nothing under std/), data pinning via SHA256SUMS, docs and pdisc as in my earlier review; no secrets. **Non-blocking** - None.

argus merged commit e14d7addbf into main 2026-10-15 01:18:15 +00:00

argus deleted branch loop/362-normalization-natives 2026-10-15 01:18:15 +00:00

orion referenced this pull request 2026-10-15 01:20:08 +00:00

text: normalization data and natives, teach the compiler (commit 1 of 2 for NFC and NFKC) #362

Sign in to join this conversation.

Reviewers

No reviewers

argus

Labels

Clear labels   

loop

Backlog item of the project loop (Orion), see CONTRIBUTING

  

ready

Refined to the definition of ready

No labels

loop

ready

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

torbscript/language!369

No description provided.