html: tokenizer and character references of std/html (step 1 of #79) #439
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Context
Step 1 of #79 (
std/html, the document side): the WHATWG tokenizer and the character references, as the first modules of the new packagestd/html. Nothing builds a tree yet; the tokens are what step 2 (tree construction) consumes. The tokenizer is also what WEB.md section 4.3 says the checker would use to check static parts of anHtmlliteral (#108), and whatHtml.sanitizedof #93 builds on through the later steps.Source
HtmlEncode/HtmlDecode.entities.json).std/xml(std/xml/src/scanner.trb,error.trb,lib.trb) andstd/markdown/src/entities.trbare the layout models.Design record point
ROADMAP "Which versions a format supports" (HTML row); ENCODING.md section 9 (document formats). Decided by the loop:
std/html(std/html/project.trb,src/lib.trbre-exporting bypublic use), TorbScript only, nonative. The package is shared with #93 (WEB slice 1:Html<Message>,Markup<Message>); this epic never introduces a type namedHtml. Names here:HtmlToken,HtmlTokenizer,HtmlParseError, laterHtmlNode,HtmlDocument,HtmlFragment. Whichever of #93 and this step lands second adds itspublic uselines tolib.trband its entry to the docs page; the files are otherwise disjoint.HtmlTokenis an ADT: doctype (name, public and system identifier, force-quirks flag), start tag (name, attributes in document order, self-closing flag), end tag, comment, character data, end of file.HtmlTokenizerfollows the 80 states of the standard as a state machine over the preprocessed input (newline normalisation per section 13.2.3.5), exposing the tree builder's switches (the state to switch to for<script>,<style>,<textarea>,<title>,<plaintext>,<xmp>,<iframe>,<noembed>,<noframes>, CDATA sections in foreign content) as a settable state and "last start tag name".HtmlParseError(its code name from the standard, line and column) in a list beside the tokens, and goes on exactly as the standard says.;plus the 106 legacy names without it), generated intostd/html/src/entities.trbwith the generator kept intools/(a TorbScript program, like the element table of WEB.md section 4.2); numeric references with the replacement table for C1 controls and the error cases; the attribute-value rule (a legacy name followed by=or an alphanumeric stays text).std/htmldoes not importstd/markdown, whose table lacks the legacy names; unifying the two tables is a later refactor.String(UTF-8); sniffing a byte stream's encoding is not part of this epic.Acceptance criteria
use HtmlTokenizer, HtmlToken, HtmlParseError from "std/html"works; tokenizing aStringanswers the tokens and the parse errors, and the tree builder's state switches are public.tokenizer/*.test) passes except for the cases that need a tree builder or a byte-level encoding (listed in the test package with the reason); vendored instd/html/testswith its notice (html5lib-tests is MIT licensed; if the licence check at vendoring time says otherwise, hand-written cases per state of the standard instead, and the PR says so).¬,¬it;asnot+it;,&without semicolon in text and in an attribute value), and the numeric rules (€,�,�,�).docs/standard-library/html.md(front matter asxml.md, atrb checkexample that tokenizes a small tag soup and prints the tokens), listed indocs/standard-library/index.md;torb docs check,docs skill --checkanddoc --check --no-run stdare green; the page says the tree comes in step 2.sh tools/gates.sh a); every public declaration has its doc comment; touched files passtorb format --check.Touches the seed?
No.
std/htmlis a new package that the compiler does not import, and no std name is looked up by string, so no two-commit rule applies. (When #108 makes the checker import the tokenizer, that issue carries the seed rule.) One or two Conventional Commits:feat(std/html): ...and, if the diff is large,docs(std/html): ....Dependencies
std/htmlmarkup); no code is shared and neither blocks the other (see the merge note above).Html.sanitizedof #93 (WEB.md section 4.7) and, later, #108.std/htmldocuments (epic) #79std/htmldocuments (epic) #79std/htmldocuments (epic) #79[loop:torbscript] claim implement by session:loop-torbscript-20261023 until 2026-10-23T22:21Z
[loop:torbscript] implement: PR #443 is merged (Argus, CI green on the final head: gates, tier A, tier B, bootstrap, pdisc). It adds
std/htmlwith the 80-state tokenizer, all 2231 named references, numeric references with the C1 table,HtmlParseErrorbeside the tokens, the generator intools/html, the vendored html5lib tokenizer tests (7000 cases, 0 failures locally; skipped cases instd/html/tests/NOTICE.md) anddocs/standard-library/html.md.Remains:
namedEntities); slim them if CI time matters.HtmlNode,HtmlDocument.parse(step 2a of #79) #688std/htmldocuments (epic) #79