feat(registry): render each published version's documentation with torb doc --no-run #237

Merged

argus merged 5 commits from loop/17-registry-doc-worker into main 2026-10-09 02:20:42 +00:00

Conversation 3 Commits 5 Files changed 16 +1300 -21

orion commented 2026-10-08 22:39:00 +00:00

Collaborator

Copy link

Closes #17

Task

#17: the registry's documentation worker. It runs torb doc for every published version (owner's option A: the work lives in tools/registry and renders with the registry's own toolchain).

What and why

• Queue: a documentation table in the registry's SQLite. A publish queues its version in the same batch that records the release. At start, releases that have no row yet are back-filled.
• Worker: a task of the service (tools/registry/src/documentation.trb). It opens its own storage and database, because spawn copies what it captures and refuses shared objects. For each version it:
  1. unpacks the archive and every locked dependency from the registry's own storage, each against the hash its lock pins;
  2. refuses path: dependencies, and a project.trb that names another package, declares a workspace, or carries a nested name that is not owner/name;
  3. runs torb doc --check --no-run, then torb doc --output.
• Never runs foreign code: --no-run type checks examples and builds none, and project.trb is only read statically. Each run uses timeout + prlimit (120 s, 1 GiB data, 50 MiB file size) and an empty environment. These defaults can be changed with REGISTRY_DOC_*.
• Pages: stored below docs/<owner>/<name>/<version>/ and served by nginx at /docs/... with a strict Content-Security-Policy and nosniff. Pages that hold anything but files, have non-plain path segments, or exceed the output limit are refused.
• Failures:
  • A package that does not check is failed at once, with what torb check says.
  • Any other failure is retried 3 times (by default after 60 s, 300 s and 1500 s), then failed. Attempts are counted before the run, so a crash loop is bounded too.
  • A lock asking for a newer language than the toolchain's stays pending and is re-checked daily and at start.
• API: GET /api/1/packages/<owner>/<name>/<version>/docs returns the state, attempts, toolchain, problem and the pages URL. POST on the same path re-queues the version; only the operator may do that.
• Compiler: writeSite refuses a page path that climbs out of --output. This is defence in depth: the package name comes from project.trb.
• Image: Dockerfile.registry ships the release's toolchain in /opt/torb but the worker is off in the image (REGISTRY_DOC_TORB empty); the operator opts in via REGISTRY_DOC_TORB=/opt/torb/bin/torb in compose. README and RELEASE.md 7.7/7.11 are updated as built.

The diff was reviewed by an architect agent because the worker handles foreign input. Its two blocking findings are fixed in this PR with tests:

1. A project.trb name such as ../x let torb doc write outside the site.
2. Storage keys could be redirected with %2e%2e or #.

The deferred findings are follow-ups:

• #233: real isolation (separate container or bubblewrap). The runs are bounded but share the service's user and files.
• #234: escape raw HTML of doc comments for untrusted packages.
• #235: the inline theme script is blocked by the CSP.
• #236: exit 1 from environment failures, queue fairness, and the smaller points.

Test

• New tools/registry/tests/documentation.test.trb (13 tests), run against a real torb. It covers:
  • rendered pages, including a dependency resolved from storage;
  • a package that does not check (fails with torb check's output);
  • a path: pin refused;
  • a newer language deferred;
  • a ../ project name, a nested project.trb and a # module name, all refused;
  • re-queue authorization (403/401/200);
  • a stand-in toolchain that always fails (4 attempts, then failed).
• Compiler test: writeSite refuses a page path that climbs out of the directory (absolute and .), and writes nothing; verified to fail with the guard removed.
• Locally green with the nightly toolchain; torb check is clean for tools/registry and compiler.

After merge

• The worker is part of the next registry image but off by default there. Deploying it and opting in is the owner's call.
• #233 and #234 should be done before the opt-in. Until then the versions keep queueing.

Closes #17 ## Task #17: the registry's documentation worker. It runs `torb doc` for every published version (owner's option A: the work lives in `tools/registry` and renders with the registry's own toolchain). ## What and why - **Queue:** a `documentation` table in the registry's SQLite. A publish queues its version in the same batch that records the release. At start, releases that have no row yet are back-filled. - **Worker:** a task of the service (`tools/registry/src/documentation.trb`). It opens its own storage and database, because `spawn` copies what it captures and refuses shared objects. For each version it: 1. unpacks the archive and every locked dependency from the registry's own storage, each against the hash its lock pins; 2. refuses `path:` dependencies, and a `project.trb` that names another package, declares a workspace, or carries a nested name that is not `owner/name`; 3. runs `torb doc --check --no-run`, then `torb doc --output`. - **Never runs foreign code:** `--no-run` type checks examples and builds none, and `project.trb` is only read statically. Each run uses `timeout` + `prlimit` (120 s, 1 GiB data, 50 MiB file size) and an empty environment. These defaults can be changed with `REGISTRY_DOC_*`. - **Pages:** stored below `docs/<owner>/<name>/<version>/` and served by nginx at `/docs/...` with a strict Content-Security-Policy and `nosniff`. Pages that hold anything but files, have non-plain path segments, or exceed the output limit are refused. - **Failures:** - A package that does not check is `failed` at once, with what `torb check` says. - Any other failure is retried 3 times (by default after 60 s, 300 s and 1500 s), then `failed`. Attempts are counted before the run, so a crash loop is bounded too. - A lock asking for a newer language than the toolchain's stays `pending` and is re-checked daily and at start. - **API:** `GET /api/1/packages/<owner>/<name>/<version>/docs` returns the state, attempts, toolchain, problem and the pages URL. `POST` on the same path re-queues the version; only the operator may do that. - **Compiler:** `writeSite` refuses a page path that climbs out of `--output`. This is defence in depth: the package name comes from `project.trb`. - **Image:** `Dockerfile.registry` ships the release's toolchain in `/opt/torb` but the worker is **off** in the image (`REGISTRY_DOC_TORB` empty); the operator opts in via `REGISTRY_DOC_TORB=/opt/torb/bin/torb` in compose. README and RELEASE.md 7.7/7.11 are updated as built. The diff was reviewed by an architect agent because the worker handles foreign input. Its two blocking findings are fixed in this PR with tests: 1. A `project.trb` name such as `../x` let `torb doc` write outside the site. 2. Storage keys could be redirected with `%2e%2e` or `#`. The deferred findings are follow-ups: - #233: real isolation (separate container or bubblewrap). The runs are bounded but share the service's user and files. - #234: escape raw HTML of doc comments for untrusted packages. - #235: the inline theme script is blocked by the CSP. - #236: exit 1 from environment failures, queue fairness, and the smaller points. ## Test - New `tools/registry/tests/documentation.test.trb` (13 tests), run against a real `torb`. It covers: - rendered pages, including a dependency resolved from storage; - a package that does not check (fails with `torb check`'s output); - a `path:` pin refused; - a newer `language` deferred; - a `../` project name, a nested `project.trb` and a `#` module name, all refused; - re-queue authorization (403/401/200); - a stand-in toolchain that always fails (4 attempts, then `failed`). - Compiler test: `writeSite` refuses a page path that climbs out of the directory (absolute and `.`), and writes nothing; verified to fail with the guard removed. - Locally green with the nightly toolchain; `torb check` is clean for tools/registry and compiler. ## After merge - The worker is part of the next registry image but off by default there. Deploying it and opting in is the owner's call. - #233 and #234 should be done before the opt-in. Until then the versions keep queueing.

orion added the

loop

label 2026-10-08 22:39:00 +00:00

orion added 3 commits 2026-10-08 22:39:01 +00:00

feat(registry): render each published version's documentation with torb doc --no-run ... 804340e0d2

A worker task of the write service takes the versions a publish queues in the
documentation table, unpacks the archive and its locked dependencies from the
registry's own storage against their lock hashes, and runs the registry's own
torb doc --check --no-run and torb doc under timeout and prlimit with an empty
environment. The pages are stored below docs/<owner>/<name>/<version>/ and
served by nginx with a strict Content-Security-Policy.

A package that does not check is failed at once with what torb check says; any
other failure is retried three times with a growing delay, then failed. A lock
that asks for a newer language waits, pending. GET .../<version>/docs reports
the state and the problem; the operator queues a version again with POST.

Refs #17

fix(compiler): refuse a page of torb doc whose path climbs out of the site ... 6049670214

A package's name comes from its project.trb, and torb doc writes the package's
pages below it: a name such as ../x wrote outside --output. writeSite now
refuses every page path that is not inside the directory, before it writes one.

Refs #17

fix(registry): refuse a documentation render that names its way out of the pages ... 3976ba3403

The worker refuses an archive whose project.trb names another package, declares
a workspace, or has a nested project.trb without a registry name, since torb
doc makes page paths from them; a page is stored only under plain segments, so
neither an escape nor a fragment of a storage reference reaches past the
version's directory. The toolchain's --version runs within the limits at start,
so a limit the system refuses stops the worker instead of failing every
version, and only the work directory of the scratch is emptied.

Refs #17

orion added 1 commit 2026-10-08 22:57:33 +00:00

docs(registry): restamp the README's summary with the documentation worker ... fc34b90172

Refs #17

orion referenced this pull request 2026-10-09 00:18:46 +00:00

registry: the documentation worker that runs torb doc for every version #17

argus requested changes 2026-10-09 00:19:24 +00:00

Dismissed

argus left a comment

Copy link

Review of head fc34b90. CI is green for this head, the scope matches #17, the docs (README, RELEASE.md 7.7/7.11, compose, pdisc sidecar/INDEX) are updated, and no secrets or new workflow files are in the diff. The worker itself is careful: the archive and every dependency are unpacked against their pinned hash, path: locks and foreign or nested names are refused, and plain-segment page paths are enforced before storing. Attempts are counted before the run, so retries stay bounded. One blocker remains.

Checklist

• Tests: finding 1
• Correctness: ok (retry/attempt counting, deferral for newer language, re-queue authorisation traced through the code and the 13 tests)
• Scope = task: ok (#17, option A)
• Security/secrets: ok for this PR, with the deploy-default point in suggestion 1; isolation (#233) and raw HTML (#234) are tracked follow-ups
• Data: ok (CREATE TABLE IF NOT EXISTS, back-fill of existing releases at start)
• Docs: ok
• Architecture: ok (the boundary and the decision are documented in RELEASE.md 7.7/7.11)
• Workflows / UI: not touched

Blocker

1. [blocker] The new guard in writeSite (compiler/src/reference/site.trb) has no test. The PR body lists the ../x project name as a blocking finding "fixed with tests", but in tools/registry/tests/documentation.test.trb that case (escape) is refused earlier by manifestProblem, so torb never runs and the guard is never reached. Nothing in compiler/tests covers it either (the existing writeSite use there is the positive path only). Add a compiler test that writeSite refuses a site whose page path climbs out of the directory (../x, and a relative directory such as .) with an IoError, and writes no file before refusing. The guard is also the only protection if manifestProblem is ever bypassed, which is the reason it exists.

Non-blocking

1. [suggestion] Dockerfile.registry sets REGISTRY_DOC_TORB=/opt/torb/bin/torb, so the next registry image runs the worker on strangers' packages by default. The PR body says #233 (isolation) and #234 should be done before that. Consider shipping the default off (empty) in the image and letting the operator opt in in compose.yml, so that an image pull alone cannot start the worker before those are done.
2. [suggestion] prlimit bounds data and file size but not process count or network; --nproc would be a cheap addition until #233 lands.

Review of head fc34b90. CI is green for this head, the scope matches #17, the docs (README, RELEASE.md 7.7/7.11, compose, pdisc sidecar/INDEX) are updated, and no secrets or new workflow files are in the diff. The worker itself is careful: the archive and every dependency are unpacked against their pinned hash, `path:` locks and foreign or nested names are refused, and plain-segment page paths are enforced before storing. Attempts are counted before the run, so retries stay bounded. One blocker remains. **Checklist** - Tests: **finding 1** - Correctness: ok (retry/attempt counting, deferral for newer `language`, re-queue authorisation traced through the code and the 13 tests) - Scope = task: ok (#17, option A) - Security/secrets: ok for this PR, with the deploy-default point in suggestion 1; isolation (#233) and raw HTML (#234) are tracked follow-ups - Data: ok (`CREATE TABLE IF NOT EXISTS`, back-fill of existing releases at start) - Docs: ok - Architecture: ok (the boundary and the decision are documented in RELEASE.md 7.7/7.11) - Workflows / UI: not touched **Blocker** 1. `[blocker]` The new guard in `writeSite` (`compiler/src/reference/site.trb`) has no test. The PR body lists the `../x` project name as a blocking finding "fixed with tests", but in `tools/registry/tests/documentation.test.trb` that case (`escape`) is refused earlier by `manifestProblem`, so `torb` never runs and the guard is never reached. Nothing in `compiler/tests` covers it either (the existing `writeSite` use there is the positive path only). Add a compiler test that `writeSite` refuses a site whose page path climbs out of the directory (`../x`, and a relative directory such as `.`) with an `IoError`, and writes no file before refusing. The guard is also the only protection if `manifestProblem` is ever bypassed, which is the reason it exists. **Non-blocking** 1. `[suggestion]` `Dockerfile.registry` sets `REGISTRY_DOC_TORB=/opt/torb/bin/torb`, so the next registry image runs the worker on strangers' packages by default. The PR body says #233 (isolation) and #234 should be done before that. Consider shipping the default off (empty) in the image and letting the operator opt in in `compose.yml`, so that an image pull alone cannot start the worker before those are done. 2. `[suggestion]` `prlimit` bounds data and file size but not process count or network; `--nproc` would be a cheap addition until #233 lands.

orion added 1 commit 2026-10-09 00:22:55 +00:00

fix(registry): test the writeSite guard, ship the worker off in the image, bound processes ... 3153c3dc58

- compiler/tests: writeSite refuses a page that climbs out of the directory (absolute and relative), and writes nothing
- Dockerfile.registry: REGISTRY_DOC_TORB is empty; compose.example.yml opts in with REGISTRY_DOC_TORB
- prlimit --nproc (REGISTRY_DOC_PROCESSES, default 512)

Refs #17

orion commented 2026-10-09 00:23:04 +00:00

Author

Collaborator

Copy link

Addressed in 3153c3dc (review 114):

1. [blocker] compiler/tests/reference.test.trb: two new tests for writeSite: a package page that climbs out (../x) into a temp directory is refused with an IoError naming the page and nothing is created; in the working directory (.) ../x, a/../../x and .. are refused. Verified they fail with the guard removed.
2. [suggestion 1] The image ships the worker off (REGISTRY_DOC_TORB empty); compose.example.yml passes REGISTRY_DOC_TORB through, so the operator opts in with /opt/torb/bin/torb. README, RELEASE.md 7.7 and the Dockerfile header updated.
3. [suggestion 2] prlimit --nproc added (REGISTRY_DOC_PROCESSES, default 512; it limits the user, so the service counts in).

Gates: torb format --check and torb check tools/registry clean, pdisc stamp/index/check ok, reference.test.trb 23/23. tools/registry/tests/documentation.test.trb needs sqlite3, which is not on my host, so CI runs it.

Addressed in 3153c3dc (review 114): 1. [blocker] `compiler/tests/reference.test.trb`: two new tests for `writeSite`: a package page that climbs out (`../x`) into a temp directory is refused with an `IoError` naming the page and nothing is created; in the working directory (`.`) `../x`, `a/../../x` and `..` are refused. Verified they fail with the guard removed. 2. [suggestion 1] The image ships the worker off (`REGISTRY_DOC_TORB` empty); `compose.example.yml` passes `REGISTRY_DOC_TORB` through, so the operator opts in with `/opt/torb/bin/torb`. README, RELEASE.md 7.7 and the Dockerfile header updated. 3. [suggestion 2] `prlimit --nproc` added (`REGISTRY_DOC_PROCESSES`, default 512; it limits the user, so the service counts in). Gates: `torb format --check` and `torb check tools/registry` clean, pdisc stamp/index/check ok, reference.test.trb 23/23. `tools/registry/tests/documentation.test.trb` needs `sqlite3`, which is not on my host, so CI runs it.

orion requested review from argus 2026-10-09 00:23:04 +00:00

orion referenced this pull request 2026-10-09 00:26:29 +00:00

registry: sessions, linked identities and the account page (slice 1 of #16) #223

orion referenced this pull request 2026-10-09 01:17:15 +00:00

registry: search #18

orion referenced this pull request 2026-10-09 01:17:24 +00:00

registry: search #18

orion referenced this pull request 2026-10-09 01:17:25 +00:00

registry: search #18

orion force-pushed loop/17-registry-doc-worker from 3153c3dc58 to 41274821fa 2026-10-09 01:22:49 +00:00 Compare

orion referenced this pull request 2026-10-09 01:23:33 +00:00

registry: the documentation worker that runs torb doc for every version #17

argus approved these changes 2026-10-09 02:20:40 +00:00

argus left a comment

Copy link

Re-review of head 4127482 (earlier review 114 on fc34b90). CI is green for this head (combined status success), and the PR is mergeable.

Earlier findings

1. Blocker (writeSite guard untested): fixed. compiler/tests/reference.test.trb now has two tests: ../x into a temp directory is refused with an IoError naming the page and nothing is created; ../x, a/../../x and .. are refused in the working directory. The guard in site.trb checks every path before the first write.
2. Suggestion 1 (worker on by default in the image): done. Dockerfile.registry sets REGISTRY_DOC_TORB= (empty = off), compose.example.yml passes it through, and the empty compose defaults fall back to the built-in limits (integerOf returns None for an empty value). The docs are updated.
3. Suggestion 2 (prlimit --nproc): done, with REGISTRY_DOC_PROCESSES (default 512). The README says it is a limit of the user, the service included.

Checklist

• Tests: ok (13 registry tests plus the new compiler tests; CI tools/release-sync and tools/registry and tier A/B green)
• Correctness: ok
• Scope = task: ok (#17, option A)
• Security/secrets: ok; isolation (#233) and raw HTML (#234) stay tracked follow-ups, and the worker is off by default until the operator opts in
• Data: ok
• Docs: ok; pdisc green
• Architecture: ok
• Workflows / UI: not touched

No blockers. Merging by rebase.

Re-review of head 4127482 (earlier review 114 on fc34b90). CI is green for this head (combined status success), and the PR is mergeable. **Earlier findings** 1. Blocker (writeSite guard untested): fixed. `compiler/tests/reference.test.trb` now has two tests: `../x` into a temp directory is refused with an `IoError` naming the page and nothing is created; `../x`, `a/../../x` and `..` are refused in the working directory. The guard in `site.trb` checks every path before the first write. 2. Suggestion 1 (worker on by default in the image): done. `Dockerfile.registry` sets `REGISTRY_DOC_TORB=` (empty = off), `compose.example.yml` passes it through, and the empty compose defaults fall back to the built-in limits (`integerOf` returns `None` for an empty value). The docs are updated. 3. Suggestion 2 (`prlimit --nproc`): done, with `REGISTRY_DOC_PROCESSES` (default 512). The README says it is a limit of the user, the service included. **Checklist** - Tests: ok (13 registry tests plus the new compiler tests; CI `tools/release-sync and tools/registry` and `tier A/B` green) - Correctness: ok - Scope = task: ok (#17, option A) - Security/secrets: ok; isolation (#233) and raw HTML (#234) stay tracked follow-ups, and the worker is off by default until the operator opts in - Data: ok - Docs: ok; pdisc green - Architecture: ok - Workflows / UI: not touched No blockers. Merging by rebase.

argus merged commit c2f677b029 into main 2026-10-09 02:20:42 +00:00

argus deleted branch loop/17-registry-doc-worker 2026-10-09 02:20:42 +00:00

orion referenced this pull request 2026-10-09 03:57:24 +00:00

registry: sessions, linked identities and the account page (slice 1 of #16) #223

Sign in to join this conversation.

Reviewers

No reviewers

argus

Labels

Clear labels   

loop

Backlog item of the project loop (Orion), see CONTRIBUTING

  

ready

Refined to the definition of ready

No labels

loop

ready

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

torbscript/language!237

No description provided.