registry: the documentation worker that runs `torb doc` for every version

orion commented

2026-10-05 21:07:10 +00:00

Collaborator

Context

The registry renders a package's reference with torb doc --json for every published version.

Source

Acceptance criteria

A worker renders each newly published version, without running its code (torb doc --check --no-run)
The pages are served by the registry
A failed render is reported on the version, not retried forever

Dependencies

None known.

Moved from the planning lists of the docs; the design record stays the specification.

## Context The registry renders a package's reference with `torb doc --json` for every published version. ## Source - [ROADMAP.md, the package manager](https://git.torb.dev/torbscript/language/src/commit/393ef2608e4a60b8ff76d8b6b408951b822d887d/docs/ROADMAP.md#L96) - [RELEASE.md section 7.11](https://git.torb.dev/torbscript/language/src/commit/393ef2608e4a60b8ff76d8b6b408951b822d887d/docs/design/RELEASE.md#L1104) ## Acceptance criteria - [ ] A worker renders each newly published version, without running its code (`torb doc --check --no-run`) - [ ] The pages are served by the registry - [ ] A failed render is reported on the version, not retried forever ## Dependencies None known. _Moved from the planning lists of the docs; the design record stays the specification._

orion added this to the Milestone 8: formatter, linter, language server, package manager milestone

2026-10-05 21:07:10 +00:00

orion added the

loop

label

2026-10-05 21:07:10 +00:00

orion commented

2026-10-06 16:16:13 +00:00

Author

Collaborator

[loop:torbscript] intake: in scope (author orion, label loop). Registry doc worker per RELEASE.md section 7.11, in tools/registry; security-relevant because it processes foreign packages (never executes them, --no-run), so opus/high worker plus architect review. Deploying the worker is outward-facing and not done by the loop. No collision with CONCEPT "Open Questions". Release freeze for 0.1.0 (2026-10-06) holds new PRs, next: refine (queued)

[loop:torbscript] intake: in scope (author orion, label loop). Registry doc worker per RELEASE.md section 7.11, in `tools/registry`; security-relevant because it processes foreign packages (never executes them, `--no-run`), so opus/high worker plus architect review. Deploying the worker is outward-facing and not done by the loop. No collision with CONCEPT "Open Questions". Release freeze for 0.1.0 (2026-10-06) holds new PRs, next: refine (queued)

orion commented

2026-10-08 12:38:20 +00:00

Author

Collaborator

[loop:torbscript] claim refine by session:loop-torbscript-20261008 until 2026-10-08T14:40:00Z

orion commented

2026-10-08 12:39:50 +00:00

Author

Collaborator

[loop:torbscript] owner decision: how the registry's documentation worker is built, served and reported (issue #17; RELEASE.md 7.7 and 7.11 leave the shape open and the result is public on packages.torb.dev).

What is settled (RELEASE.md 7.7, 7.11, torb-doc.md): the page of a version is torb doc of it; it runs none of the package's code (torb doc --check --no-run); the worker runs under time and memory limits; the worker is "the same program, or a second process, with the toolchain of every released version"; the files are static, served beside /index and /archives.

What is not settled (each is a choice a later reader of the registry depends on):

Which toolchain renders a version. 7.7 says "the toolchain the package's language line names", but the language line is a minimum version (RELEASE.md 3, no editions). The registry image carries one toolchain (the build stage's), not "every released version".
Process shape. In-process task of the write service, or a second process/container.
Output and URL. torb doc --output writes a finished static site (index.html, search.js, reference.json, ...); 7.7 also says "rendered by the same site generator as torb.dev", which has no reader of the JSON model yet (RELEASE.md 6 calls that a follow-up). 7.7 names /<owner>/<name>/<version>/docs, the 7.11 diagram /docs.
Where a failed render is reported (the issue's third criterion): the signed, append-only index (a format change every client sees), or the write service's database plus a read endpoint.
Limits and retry numbers (time, memory, output size, attempts).

Options.

A (recommended). (1) Render with the registry's own toolchain, the newest released one, because 1.x is strictly compatible (a package whose language line is newer than the toolchain is rendered as soon as the registry image carries it; until then it is pending, not failed). (2) In-process: a queue table in the existing SQLite database, one worker task at a time, torb doc --check --no-run then torb doc --output run as a child process through src/program.trb's pipes, with ulimit-style time and memory limits (defaults 120 s, 1 GiB, 50 MiB of output; configurable by REGISTRY_DOC_*). (3) Use torb doc's own static site, written to <storage>/docs/<owner>/<name>/<version>/ through the storage layer and written whole or not at all (rename of a finished directory), served at packages.torb.dev/docs/<owner>/<name>/<version>/; the JSON-rendered site-generator look is a later issue. (4) Status in the database (pending, done, failed with the first lines of the problem list and the attempt count), exposed read-only as GET /api/1/packages/<owner>/<name>/<version>/docs and linked from the version's page; the index is not touched. (5) A failure retries at most 3 times with growing delay, then stays failed until a newer toolchain is deployed (operator re-queue endpoint) - never retried forever. Deployment of the new image stays the owner's.
B. As A, but a second container with a pinned toolchain per released language value (the literal reading of 7.7): more moving parts (a toolchain store fed by release-sync, a scheduler picking the toolchain) and a surface that grows with every release.
C. As A, but the failure goes into the signed index as a record kind (docs-failed), so mirrors and clients see it: a public protocol change, hard to take back, and a render failure is operational data, not a fact about the release.
D. Split the work first and decide shape per slice; no code until all of 1-5 are answered.

Split after the answer (dependency-linked issues, label loop): (i) the queue, status table and render step in tools/registry with tests; (ii) the status endpoint and the served path; (iii) docs: README "Not built yet" and RELEASE.md 7.7/7.11 updated to the chosen shape. Deploying the worker, the image and the server configuration are not loop work.

Recommendation. Option A. Say "go A", "go B", "go C" or name the changes.

[loop:torbscript] owner decision: how the registry's documentation worker is built, served and reported (issue #17; RELEASE.md 7.7 and 7.11 leave the shape open and the result is public on packages.torb.dev). **What is settled** ([RELEASE.md 7.7](https://git.torb.dev/torbscript/language/src/commit/4c6bbc403d419bc769ed3ed387d3d25e629d45ba/docs/design/RELEASE.md#L1044), [7.11](https://git.torb.dev/torbscript/language/src/commit/4c6bbc403d419bc769ed3ed387d3d25e629d45ba/docs/design/RELEASE.md#L1104), [torb-doc.md](https://git.torb.dev/torbscript/language/src/commit/4c6bbc403d419bc769ed3ed387d3d25e629d45ba/docs/tooling/torb-doc.md)): the page of a version is `torb doc` of it; it runs none of the package's code (`torb doc --check --no-run`); the worker runs under time and memory limits; the worker is "the same program, or a second process, with the toolchain of every released version"; the files are static, served beside `/index` and `/archives`. **What is not settled** (each is a choice a later reader of the registry depends on): 1. *Which toolchain renders a version.* 7.7 says "the toolchain the package's `language` line names", but the `language` line is a *minimum* version ([RELEASE.md 3](https://git.torb.dev/torbscript/language/src/commit/4c6bbc403d419bc769ed3ed387d3d25e629d45ba/docs/design/RELEASE.md#L161), no editions). The registry image carries one toolchain (the build stage's), not "every released version". 2. *Process shape.* In-process task of the write service, or a second process/container. 3. *Output and URL.* `torb doc --output` writes a finished static site (`index.html`, `search.js`, `reference.json`, ...); 7.7 also says "rendered by the same site generator as torb.dev", which has no reader of the JSON model yet (RELEASE.md 6 calls that a follow-up). 7.7 names `/<owner>/<name>/<version>/docs`, the 7.11 diagram `/docs`. 4. *Where a failed render is reported* (the issue's third criterion): the signed, append-only index (a format change every client sees), or the write service's database plus a read endpoint. 5. *Limits and retry numbers* (time, memory, output size, attempts). **Options.** - **A (recommended).** (1) Render with the registry's own toolchain, the newest released one, because 1.x is strictly compatible (a package whose `language` line is newer than the toolchain is rendered as soon as the registry image carries it; until then it is `pending`, not failed). (2) In-process: a queue table in the existing SQLite database, one worker task at a time, `torb doc --check --no-run` then `torb doc --output` run as a child process through `src/program.trb`'s pipes, with `ulimit`-style time and memory limits (defaults 120 s, 1 GiB, 50 MiB of output; configurable by `REGISTRY_DOC_*`). (3) Use `torb doc`'s own static site, written to `<storage>/docs/<owner>/<name>/<version>/` through the storage layer and written whole or not at all (rename of a finished directory), served at `packages.torb.dev/docs/<owner>/<name>/<version>/`; the JSON-rendered site-generator look is a later issue. (4) Status in the database (`pending`, `done`, `failed` with the first lines of the problem list and the attempt count), exposed read-only as `GET /api/1/packages/<owner>/<name>/<version>/docs` and linked from the version's page; the index is not touched. (5) A failure retries at most 3 times with growing delay, then stays `failed` until a newer toolchain is deployed (operator re-queue endpoint) - never retried forever. Deployment of the new image stays the owner's. - **B.** As A, but a second container with a pinned toolchain per released `language` value (the literal reading of 7.7): more moving parts (a toolchain store fed by release-sync, a scheduler picking the toolchain) and a surface that grows with every release. - **C.** As A, but the failure goes into the signed index as a record kind (`docs-failed`), so mirrors and clients see it: a public protocol change, hard to take back, and a render failure is operational data, not a fact about the release. - **D.** Split the work first and decide shape per slice; no code until all of 1-5 are answered. **Split after the answer** (dependency-linked issues, label `loop`): (i) the queue, status table and render step in `tools/registry` with tests; (ii) the status endpoint and the served path; (iii) docs: README "Not built yet" and RELEASE.md 7.7/7.11 updated to the chosen shape. Deploying the worker, the image and the server configuration are not loop work. **Recommendation.** Option A. Say "go A", "go B", "go C" or name the changes.

orion commented

2026-10-08 12:40:13 +00:00

Author

Collaborator

Question q1 for torben (decision, blocking): How should the registry's documentation worker be built, served and reported?

v: 1
id: q1
to: torben
kind: decision
blocking: true
question: "How should the registry's documentation worker be built, served and reported? Details in the owner decision comment above (toolchain choice, process shape, output and URL, where a failed render is reported, limits)."
options:
  - id: a
    label: "A: newest toolchain, in-process worker, DB status"
    detail: Registry's own newest toolchain; queue in SQLite; torb doc static site served under /docs/<owner>/<name>/<version>/; failures in the database with a read endpoint, max 3 retries
  - id: b
    label: "B: second container, toolchain per language value"
    detail: Literal reading of 7.7; needs a toolchain store fed by release-sync and a scheduler
  - id: c
    label: "C: failure recorded in the signed index"
    detail: As A, but a docs-failed index record; public protocol change
  - id: d
    label: "D: decide per slice"
    detail: No code until every point is answered
recommendation: a
asked_by: orion

**Question q1** for torben (decision, blocking): How should the registry's documentation worker be built, served and reported? ```hos-question v: 1 id: q1 to: torben kind: decision blocking: true question: "How should the registry's documentation worker be built, served and reported? Details in the owner decision comment above (toolchain choice, process shape, output and URL, where a failed render is reported, limits)." options: - id: a label: "A: newest toolchain, in-process worker, DB status" detail: Registry's own newest toolchain; queue in SQLite; torb doc static site served under /docs/<owner>/<name>/<version>/; failures in the database with a read endpoint, max 3 retries - id: b label: "B: second container, toolchain per language value" detail: Literal reading of 7.7; needs a toolchain store fed by release-sync and a scheduler - id: c label: "C: failure recorded in the signed index" detail: As A, but a docs-failed index record; public protocol change - id: d label: "D: decide per slice" detail: No code until every point is answered recommendation: a asked_by: orion ```

orion commented

2026-10-08 12:40:13 +00:00

Author

Collaborator

[loop:torbscript] refine: owner decision filed (q1), next: waiting (owner decision q1). Checked first: no answer by torben, no [owner-answer], none recorded. Not ready: the toolchain choice, process shape, served URL and failure-report surface are open in RELEASE.md 7.7/7.11, and the result is public on packages.torb.dev. Size after the answer: three dependency-linked issues (queue/status/render in tools/registry; status endpoint and served path; docs update), implementer opus/high plus architect review (foreign input), seed impact none expected. Refine worker usage: sonnet, about 90k tokens.

[loop:torbscript] refine: owner decision filed (q1), next: waiting (owner decision q1). Checked first: no answer by torben, no `[owner-answer]`, none recorded. Not ready: the toolchain choice, process shape, served URL and failure-report surface are open in RELEASE.md 7.7/7.11, and the result is public on packages.torb.dev. Size after the answer: three dependency-linked issues (queue/status/render in `tools/registry`; status endpoint and served path; docs update), implementer opus/high plus architect review (foreign input), seed impact none expected. Refine worker usage: sonnet, about 90k tokens.

orion commented

2026-10-08 14:36:22 +00:00

Author

Collaborator

[loop:torbscript] owner decision recorded (torben via WhatsApp, 2026-10-08 16:35 Europe/Berlin): "Okay" (option A).

Resolution: the registry's own newest toolchain, queue in SQLite, in-process worker, torb doc static site under /docs/<owner>/<name>/<version>/, failures in the database with a read endpoint, max 3 retries. Labelled ready.

[loop:torbscript] owner decision recorded (torben via WhatsApp, 2026-10-08 16:35 Europe/Berlin): "Okay" (option A). Resolution: the registry's own newest toolchain, queue in SQLite, in-process worker, `torb doc` static site under `/docs/<owner>/<name>/<version>/`, failures in the database with a read endpoint, max 3 retries. Labelled `ready`.

orion added the

ready

label

2026-10-08 14:36:22 +00:00

orion commented

2026-10-08 17:48:53 +00:00

Author

Collaborator

[question-closed] q=q1 reason=answered-elsewhere

orion commented

2026-10-08 22:04:39 +00:00

Author

Collaborator

[loop:torbscript] claim implement by session:loop-torbscript-20261009 until 2026-10-09T04:04Z

orion referenced this issue from a commit

2026-10-08 22:38:25 +00:00

feat(registry): render each published version's documentation with torb doc --no-run

orion referenced this issue from a commit

2026-10-08 22:38:25 +00:00

fix(compiler): refuse a page of torb doc whose path climbs out of the site

orion referenced this issue from a commit

2026-10-08 22:38:25 +00:00

fix(registry): refuse a documentation render that names its way out of the pages

orion referenced this issue

2026-10-08 22:38:44 +00:00

registry: isolate the documentation worker's runs from the service #233

orion referenced this issue

2026-10-08 22:38:45 +00:00

torb doc: escape raw HTML of doc comments when rendering an untrusted package #234

orion referenced this issue

2026-10-08 22:38:45 +00:00

torb doc: move the inline head script into a file so a strict CSP allows it #235

orion referenced this issue

2026-10-08 22:38:45 +00:00

registry: tell a package's fault from the environment's in documentation renders #236

orion referenced this issue from a pull request that will close it,

2026-10-08 22:39:01 +00:00

feat(registry): render each published version's documentation with torb doc --no-run #237

orion referenced this issue from a commit

2026-10-08 22:57:33 +00:00

docs(registry): restamp the README's summary with the documentation worker

orion commented

2026-10-09 00:18:46 +00:00

Author

Collaborator

[loop:torbscript] implement: PR #237, CI success, next: review (waiting on the merge gate argus)

Token usage: about 500k (implementer) plus about 200k (architect review and re-review).

Follow-ups filed: #233 (render isolation, before production), #234 (escape raw HTML in torb doc, before production), #235 (inline head script vs CSP), #236 (smaller worker hardening and fairness items).

[loop:torbscript] implement: PR https://git.torb.dev/torbscript/language/pulls/237, CI success, next: review (waiting on the merge gate argus) Token usage: about 500k (implementer) plus about 200k (architect review and re-review). Follow-ups filed: #233 (render isolation, before production), #234 (escape raw HTML in `torb doc`, before production), #235 (inline head script vs CSP), #236 (smaller worker hardening and fairness items).

orion referenced this issue

2026-10-09 00:21:21 +00:00

parallel: Window and windows (CONCURRENCY slice F, gap 6) #4

orion referenced this issue from a commit

2026-10-09 00:22:55 +00:00

fix(registry): test the writeSite guard, ship the worker off in the image, bound processes

orion referenced this issue

2026-10-09 00:26:29 +00:00

registry: sessions, linked identities and the account page (slice 1 of #16) #223

orion referenced this issue

2026-10-09 01:17:15 +00:00

registry: search #18

orion referenced this issue

2026-10-09 01:17:24 +00:00

registry: search #18

orion referenced this issue from a commit

2026-10-09 01:22:49 +00:00

fix(registry): test the writeSite guard, ship the worker off in the image, bound processes

orion commented

2026-10-09 01:23:33 +00:00

Author

Collaborator

[loop:torbscript] review: CI on PR #237 died twice with a runner "context canceled" (dind docker.sock) after the tests had passed; there is no rerun endpoint, so the branch head was re-created with identical content (new head 4127482) to retrigger forge CI. next: waiting (CI running, then the merge gate argus)

[loop:torbscript] review: CI on PR #237 died twice with a runner "context canceled" (dind docker.sock) after the tests had passed; there is no rerun endpoint, so the branch head was re-created with identical content (new head 4127482) to retrigger forge CI. next: waiting (CI running, then the merge gate argus)