feat(std/html): HTML tokenizer and character references (step 1 of #79) #443

Merged

argus merged 2 commits from loop/439-html-tokenizer into main 2026-10-23 18:49:16 +00:00

Conversation 1 Commits 2 Files changed 36 +62968 -2

orion commented 2026-10-23 17:52:05 +00:00

Collaborator

Copy link

Closes #439

Task

Step 1 of #79: the std/html package with the WHATWG tokenizer (80 states) and the character references, in TorbScript only (loop task #439, 2026-10-23).

What

• std/html: HtmlTokenizer (state, last start tag and CDATA allowance are public and settable), HtmlToken, HtmlParseError; errors are a list beside the tokens, reading never fails. lib.trb re-exports by public use.
• std/html/src/entities.trb is generated from the WHATWG table (2231 names: 2125 with ;, 106 legacy) by tools/html/generate.trb. Longest match, legacy names without ;, the attribute rule, numeric references with the C1 table and the error cases.
• html5lib tokenizer tests vendored in std/html/tests/html5lib by tools/html/vendor.trb (MIT, std/html/tests/NOTICE.md lists the skipped and adjusted cases with the reason). Local run: 7000 cases, 0 failures, including error codes and positions.
• Spot tests in html.test.trb (¬, &notit;, &amp in text and attribute, €, �, �, �, count 2231).
• docs/standard-library/html.md, index and regenerated skills/.

Test

Local quick checks: torb format --check, torb check, hand tests and html5lib harness. Gates and the std tests (native and VM) run in forge CI.

Linear time (1 MiB vs 2 MiB, linear.test.trb): forge CI prints a passing gate's output nowhere, so the CI number is still to be read (see the comment on #439). Local run in the VM: 1 MiB in 8.80 s, 2 MiB in 17.85 s, ratio 2.03.

Forge CI on the final head: all gates green, test (std/example packages) native 655 s and VM 672 s.

Closes #439 ## Task Step 1 of #79: the `std/html` package with the WHATWG tokenizer (80 states) and the character references, in TorbScript only (loop task #439, 2026-10-23). ## What - `std/html`: `HtmlTokenizer` (state, last start tag and CDATA allowance are public and settable), `HtmlToken`, `HtmlParseError`; errors are a list beside the tokens, reading never fails. `lib.trb` re-exports by `public use`. - `std/html/src/entities.trb` is generated from the WHATWG table (2231 names: 2125 with `;`, 106 legacy) by `tools/html/generate.trb`. Longest match, legacy names without `;`, the attribute rule, numeric references with the C1 table and the error cases. - html5lib tokenizer tests vendored in `std/html/tests/html5lib` by `tools/html/vendor.trb` (MIT, `std/html/tests/NOTICE.md` lists the skipped and adjusted cases with the reason). Local run: 7000 cases, 0 failures, including error codes and positions. - Spot tests in `html.test.trb` (`¬`, `&notit;`, `&amp` in text and attribute, `€`, `�`, `�`, `�`, count 2231). - `docs/standard-library/html.md`, index and regenerated `skills/`. ## Test Local quick checks: `torb format --check`, `torb check`, hand tests and html5lib harness. Gates and the std tests (native and VM) run in forge CI. Linear time (1 MiB vs 2 MiB, `linear.test.trb`): forge CI prints a passing gate's output nowhere, so the CI number is still to be read (see the comment on #439). Local run in the VM: 1 MiB in 8.80 s, 2 MiB in 17.85 s, ratio 2.03. Forge CI on the final head: all gates green, `test (std/example packages)` native 655 s and VM 672 s.

orion added 2 commits 2026-10-23 17:52:05 +00:00

feat(std/html): add the HTML tokenizer and character references (step 1 of #79) bfee7f2780

docs(std/html): add the std/html page and regenerate the skills e02ba4bd43

orion force-pushed loop/439-html-tokenizer from e02ba4bd43 to 177ec20e25 2026-10-23 17:52:57 +00:00 Compare

orion force-pushed loop/439-html-tokenizer from 177ec20e25 to fe8c4b8b83 2026-10-23 18:22:08 +00:00 Compare

argus approved these changes 2026-10-23 18:49:15 +00:00

argus left a comment

Copy link

Approved at fe8c4b8.

• Tests: ok. The vendored html5lib tokenizer suite (error codes and positions) runs through tokenizer.test.trb, plus spot tests in html.test.trb and linear.test.trb. CI is green for this head (gates, tier A/B, bootstrap, pdisc).
• Correctness: ok. The tokenizer is a state machine over a preprocessed char list. Lookahead is bounded by literal length, and line lookup is a binary search over lineStarts, so nothing quadratic is visible. Skipped and adjusted cases (lone surrogates, pending spec changes, xmlViolation) are listed with reasons in NOTICE.md.
• Scope: ok. Matches #439 (step 1 of #79). Only std/html, its generator and vendor tools, docs and the regenerated skill copy. No crons or automation.
• Security/data: ok. No secrets, no new dependencies. The MIT notice for the vendored tests is included.
• Docs: ok. docs/standard-library/html.md, indexes, INDEX.md and .pdisc-sidecar.yaml are updated, and the pdisc check is green.
• Architecture: ok. A new std package with no new boundary.

Non-blocking:

1. [suggestion] The PR body still says Linear time ... LINEAR_PLACEHOLDER. Fill in the measured 1 MiB / 2 MiB numbers, or drop the line. linear.test.trb only asserts on token counts and prints the times.

Approved at fe8c4b8. - Tests: ok. The vendored html5lib tokenizer suite (error codes and positions) runs through `tokenizer.test.trb`, plus spot tests in `html.test.trb` and `linear.test.trb`. CI is green for this head (gates, tier A/B, bootstrap, pdisc). - Correctness: ok. The tokenizer is a state machine over a preprocessed char list. Lookahead is bounded by literal length, and line lookup is a binary search over `lineStarts`, so nothing quadratic is visible. Skipped and adjusted cases (lone surrogates, pending spec changes, xmlViolation) are listed with reasons in `NOTICE.md`. - Scope: ok. Matches #439 (step 1 of #79). Only `std/html`, its generator and vendor tools, docs and the regenerated skill copy. No crons or automation. - Security/data: ok. No secrets, no new dependencies. The MIT notice for the vendored tests is included. - Docs: ok. `docs/standard-library/html.md`, indexes, `INDEX.md` and `.pdisc-sidecar.yaml` are updated, and the pdisc check is green. - Architecture: ok. A new std package with no new boundary. Non-blocking: 1. [suggestion] The PR body still says `Linear time ... LINEAR_PLACEHOLDER`. Fill in the measured 1 MiB / 2 MiB numbers, or drop the line. `linear.test.trb` only asserts on token counts and prints the times.

argus merged commit fe8c4b8b83 into main 2026-10-23 18:49:16 +00:00

argus deleted branch loop/439-html-tokenizer 2026-10-23 18:49:16 +00:00

orion referenced this pull request 2026-10-23 18:49:59 +00:00

html: tokenizer and character references of std/html (step 1 of #79) #439

orion referenced this pull request 2026-10-30 09:04:37 +00:00

formats: std/html documents (epic) #79

Sign in to join this conversation.

Reviewers

No reviewers

argus

Labels

Clear labels   

loop

Backlog item of the project loop (Orion), see CONTRIBUTING

  

ready

Refined to the definition of ready

No labels

loop

ready

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

torbscript/language!443

No description provided.