package manager: fetch and upload through std/http instead of curl/wget (needs #30) #333

Closed
opened 2026-10-12 13:15:50 +00:00 by orion · 8 comments
Collaborator

Context

Split from #30. The package manager fetches and uploads through curl, else wget (compiler/src/package/fetch.trb, compiler/src/package/upload.trb, RELEASE.md section 7.13 "How the client fetches") because torb could not wait for a task. #30 adds a synchronous wait (Task.runToEnd()); this issue moves the client onto std/http.

Depends on

  • #30 merged and the seed refreshed from it (the compiler's own source may only call runToEnd() once the seed knows it; CLAUDE.md "Seed and breaking changes", two commits). The seed refresh is the owner's, so the loop treats this issue as waiting until the published seed contains the native.

Acceptance criteria

  • fetch.trb downloads an index and an archive with std/http's Client (redirects followed within the registry's origin policy, HTTPS with the platform's trust) and writes the bytes with File.writeBytesAtomically; a 404 of an index is still "the registry does not have the package"
  • upload.trb (torb publish, login, owner, yank, deprecate requests) uses std/http the same way
  • TORB_FETCH and every curl/wget spawn in compiler/src/package/ are gone; a machine without curl and wget can torb add, install and publish (a test runs the client against a local registry served by std/http with PATH emptied of both tools)
  • --offline, mirrors, signature and tree-hash checks behave unchanged (existing tests stay green)
  • torb run and torb test still start the VM scheduler (no top-level task is introduced)
  • RELEASE.md section 7.13 table row, the docs-comments of fetch.trb/upload.trb/store.trb and docs/tooling/torb-add.md describe the new behaviour; skills/ regenerated
  • torb upgrade (compiler/src/cli/upgrade.trb) is out of scope here unless the change is mechanical; if left on curl, its doc comment keeps saying why

Design record

docs/design/RELEASE.md sections 7.4, 7.5, 7.13; docs/design/NETWORK.md (client). Seed impact: none beyond the dependency above (no new native, no name the compiler looks up by string).

Size

One PR, one Conventional Commit (feat(package): fetch and upload through std/http).

## Context Split from #30. The package manager fetches and uploads through `curl`, else `wget` (`compiler/src/package/fetch.trb`, `compiler/src/package/upload.trb`, `RELEASE.md` section 7.13 "How the client fetches") because `torb` could not wait for a task. #30 adds a synchronous wait (`Task.runToEnd()`); this issue moves the client onto `std/http`. ## Depends on - #30 merged **and the seed refreshed from it** (the compiler's own source may only call `runToEnd()` once the seed knows it; `CLAUDE.md` "Seed and breaking changes", two commits). The seed refresh is the owner's, so the loop treats this issue as waiting until the published seed contains the native. ## Acceptance criteria - [ ] `fetch.trb` downloads an index and an archive with `std/http`'s `Client` (redirects followed within the registry's origin policy, HTTPS with the platform's trust) and writes the bytes with `File.writeBytesAtomically`; a 404 of an index is still "the registry does not have the package" - [ ] `upload.trb` (`torb publish`, `login`, `owner`, `yank`, `deprecate` requests) uses `std/http` the same way - [ ] `TORB_FETCH` and every `curl`/`wget` spawn in `compiler/src/package/` are gone; a machine without `curl` and `wget` can `torb add`, `install` and `publish` (a test runs the client against a local registry served by `std/http` with `PATH` emptied of both tools) - [ ] `--offline`, mirrors, signature and tree-hash checks behave unchanged (existing tests stay green) - [ ] `torb run` and `torb test` still start the VM scheduler (no top-level task is introduced) - [ ] `RELEASE.md` section 7.13 table row, the docs-comments of `fetch.trb`/`upload.trb`/`store.trb` and `docs/tooling/torb-add.md` describe the new behaviour; `skills/` regenerated - [ ] `torb upgrade` (`compiler/src/cli/upgrade.trb`) is out of scope here unless the change is mechanical; if left on `curl`, its doc comment keeps saying why ## Design record `docs/design/RELEASE.md` sections 7.4, 7.5, 7.13; `docs/design/NETWORK.md` (client). Seed impact: none beyond the dependency above (no new native, no name the compiler looks up by string). ## Size One PR, one Conventional Commit (`feat(package): fetch and upload through std/http`).
Author
Collaborator

[loop:torbscript] refine: split from #30, not ready, next: waiting (dependency #30 merged and seed refreshed)

[loop:torbscript] refine: split from #30, not ready, next: waiting (dependency #30 merged and seed refreshed)
Author
Collaborator

[loop:torbscript] claim refine by session:loop-torbscript-20261029 until 2026-10-29T10:20Z

[loop:torbscript] claim refine by session:loop-torbscript-20261029 until 2026-10-29T10:20Z
Author
Collaborator

[loop:torbscript] decided by the loop: the dependency is met on main, and the shape of the migration is as follows (verified against origin/main @ 4b59fae0).

Dependency check

  • #30 landed as 7bfb4088 (feat(task): run a task to its end from synchronous code, 2026-10-12): native fn runToEnd(): Result<Value, Cancelled> is declared in std/task/src/lib.trb, documented in docs/standard-library/task.md, and tests/conformance/task-run-to-end-http.trb runs a std/http server and client from a top level without await().
  • The published seeds contain it: seeds.txt lists 4b3045d31334 (2026-10-29) and acf8510278d7 first; both have 7bfb4088 as an ancestor and declare runToEnd in std/task/src/lib.trb. CI bootstraps from the newest published seed, so the two-commit rule of CLAUDE.md is satisfied and no further seed refresh is needed. The "waiting on a seed refresh" condition is cleared.
  • The second reason RELEASE.md 7.13 gives for curl (name resolution crashing on Windows before Winsock is loaded) was fixed on 2026-09-27 and checked on 2026-09-28; it is no reason against std/http any more.

What std/http offers (checked in std/http/src/client.trb, pool.trb, message.trb)

  • Client(maximumConnections, timeout, redirects, tls, resolver); methods get, post, send(Method.Put | Delete | Post ...); Body.from(Bytes) sends a known Content-Length; response.body.bytes(limit:) collects a body up to a limit (default 16 MiB); Headers.set.
  • TLS: https over mbedTLS with the platform's trust (crypt32 on Windows, $SSL_CERT_FILE or the system bundle on POSIX), the URL's host as the certificate name.
  • Redirects: RedirectPolicy.Never | SameHost(limit) | AnyHost(limit); https to http is never followed; AnyHost strips Authorization, Cookie, Proxy-Authorization when the host changes; a redirect that would have to resend a streamed body is returned as the response.
  • Auth: caller-supplied headers only (a bearer token is just a header, kept in memory instead of a header file).
  • Gaps: no HTTP_PROXY/HTTPS_PROXY/NO_PROXY support, HTTP/1.1 only, no automatic content decoding.

Decisions

  1. One synchronous helper, one task per request. A new compiler/src/package/network.trb holds the only Task.runToEnd() call site: it runs one async function that builds a Client inside the task (a Client is a shared type that stays with the task that made it), sends, reads the body with a limit and returns plain values. A fresh Client per request: a package resolution makes a handful of requests, and no connection or pool state has to survive between two scheduler runs. fetch.trb and upload.trb call only this helper. Nothing else in compiler/src calls runToEnd(), and no top level gets an await() (main.trb stays a synchronous top level, so torb run/torb test still start the VM scheduler).
  2. Redirects. Fetches (index, config.trb, owner file, archive, archive: URL, mirrors alike) use RedirectPolicy.AnyHost(5): no credential is ever sent on a fetch and every byte is checked (index signatures, tree hash), so a CDN hop is safe, and it keeps what curl -L did. https to http is never followed (stricter than curl -L). Requests that carry a token (PUT upload, login, owner, yank, deprecate, the trusted-publishing exchange, the CI identity-token GET) use RedirectPolicy.Never, as the old curl call had no -L: a 3xx is reported as a refusal (<url> answered 301, redirecting to <Location>, printable), so a token is never re-sent anywhere.
  3. Status mapping unchanged. 404 and 410 of a fetch are "not found" (Ok(false), the index case "the registry does not have the package"); any other non-2xx or an unreachable host is a Fail naming the URL and the status/cause in printable text. A request with a token returns the status and body as a Reply as before (a refusal is a Reply, not an error).
  4. Limits (numbers). Head deadline Client(timeout: Some(30.seconds())) (curl had none; a hung registry should not hang torb add); fetch bodies bytes(limit: 16 MiB) (the archive cap is 10 MiB compressed); API replies bytes(limit: 1 MiB); a non-UTF-8 API reply reads as empty text as before.
  5. Token handling. The token travels as a header value in memory; no header file, answer file, 0700 upload directory or 0600 helper file is needed, so uploadDirectory, writePrivate, RequestFiles, requestArguments and the work parameters go away, exchangedToken's send callback receives the JSON text instead of a file path, and request(...) keeps its method, url, headers, body shape for cli/registry.trb. The rule "token only over https: or loopback http:, never to a mirror" (tokenOf, credentials.trb) is unchanged.
  6. TORB_FETCH is removed, as the acceptance criteria say (documented in torb-install.md, torb-publish.md; also set in tools/registry/tests/commands.test.trb and owner-step-up.test.trb, which lose it). file: registries, --offline, mirror order, signature and tree-hash checks are untouched (fetchFirst keeps its logic and only fetchUrl changes; the download.part file may stay).
  7. Known limitation, documented, not a blocker: std/http does not read proxy variables, which curl/wget honoured implicitly. docs/tooling/torb-install.md states "the client does not use a proxy yet". A proxy follow-up is filed only when someone needs it.
  8. Link impact is accepted: torb will link std/http, std/tls, std/dns and the vendored mbedTLS (compiled only where reached). No feature flag. The implementer reports the binary size and bootstrap time change from the forge CI run in the PR body; a red gate is fixed on the PR.

Acceptance criteria (replace those in the issue body)

  • network.trb (helper above) is the only runToEnd() caller in compiler/src; fetch.trb downloads index files, config.trb, owner files and archives through it and writes them with File.writeBytesAtomically; 404/410 of an index is still "the registry does not have the package"; redirects as in decision 2
  • upload.trb (torb publish, login, owner, yank, deprecate, trusted publishing) sends through the same helper with Never redirects; the token appears in no process argument and in no file
  • No curl/wget/TORB_FETCH in compiler/src/package/; a machine without them can torb add, install, publish (a test runs the torb binary as a subprocess against a loopback registry served by std/http, with PATH pointing at a directory that has neither tool, in the style of tools/registry/tests/client.test.trb)
  • Tests for the helper: 404 and 410 on fetch, a 500, a same-host and a cross-host redirect on fetch, a redirect on a token request is a refusal that sends the token nowhere, an https-to-http redirect is not followed, a body over the limit, an unreachable host names the URL. If test bodies turn out to run inside a task (so runToEnd() would panic), put these in tests/conformance/ programs with a synchronous top level, as task-run-to-end-http.trb does
  • --offline, mirrors, signature and tree-hash checks behave unchanged; the existing suites (compiler/tests/packages.test.trb, registry-commands.test.trb, tools/registry/tests/*) stay green after their curl-specific parts (the fake-curl argument log, requestArguments, uploadDirectory) are replaced by the new tests
  • torb run and torb test still start the VM scheduler (no top-level task is introduced)
  • RELEASE.md 7.13 "How the client fetches" row, the module comments of fetch.trb/upload.trb/store.trb ("everything here is synchronous because ...") and docs/tooling/torb-install.md, torb-add.md, torb-publish.md, torb-login.md describe the new behaviour (including the proxy limitation); skills/ regenerated
  • torb upgrade and the Url source of torb template (compiler/src/cli/upgrade.trb, cli/template.trb) stay on curl/wget unless the change is mechanical with the new helper; if left, their doc comments keep saying why
  • PR body reports the link-size and bootstrap-time change (decision 8)

Seed and size

  • Seed impact: none (no new native, no name the compiler looks up by string; runToEnd is already in the published seed). If the bootstrap in CI fails because std/http at main uses a form the newest published seed cannot read, that is a new wait on a seed publication, not a design change.
  • Size: one PR, one Conventional Commit feat(package): fetch and upload through std/http (a second docs(package): ... commit is fine if the docs and skills/ regeneration are easier to review apart). Not an epic.
  • Design record: docs/design/RELEASE.md 7.4, 7.5, 7.13; docs/design/NETWORK.md (client, redirects); docs/standard-library/task.md (runToEnd). ROADMAP: "The package manager" bullet.
[loop:torbscript] decided by the loop: the dependency is met on `main`, and the shape of the migration is as follows (verified against `origin/main` @ 4b59fae0). ## Dependency check - #30 landed as `7bfb4088` (`feat(task): run a task to its end from synchronous code`, 2026-10-12): `native fn runToEnd(): Result<Value, Cancelled>` is declared in `std/task/src/lib.trb`, documented in `docs/standard-library/task.md`, and `tests/conformance/task-run-to-end-http.trb` runs a `std/http` server and client from a top level without `await()`. - The published seeds contain it: `seeds.txt` lists `4b3045d31334` (2026-10-29) and `acf8510278d7` first; both have `7bfb4088` as an ancestor and declare `runToEnd` in `std/task/src/lib.trb`. CI bootstraps from the newest published seed, so the two-commit rule of `CLAUDE.md` is satisfied and no further seed refresh is needed. The "waiting on a seed refresh" condition is cleared. - The second reason `RELEASE.md` 7.13 gives for `curl` (name resolution crashing on Windows before Winsock is loaded) was fixed on 2026-09-27 and checked on 2026-09-28; it is no reason against `std/http` any more. ## What `std/http` offers (checked in `std/http/src/client.trb`, `pool.trb`, `message.trb`) - `Client(maximumConnections, timeout, redirects, tls, resolver)`; methods `get`, `post`, `send(Method.Put | Delete | Post ...)`; `Body.from(Bytes)` sends a known `Content-Length`; `response.body.bytes(limit:)` collects a body up to a limit (default 16 MiB); `Headers.set`. - TLS: `https` over mbedTLS with the platform's trust (crypt32 on Windows, `$SSL_CERT_FILE` or the system bundle on POSIX), the URL's host as the certificate name. - Redirects: `RedirectPolicy.Never | SameHost(limit) | AnyHost(limit)`; `https` to `http` is never followed; `AnyHost` strips `Authorization`, `Cookie`, `Proxy-Authorization` when the host changes; a redirect that would have to resend a streamed body is returned as the response. - Auth: caller-supplied headers only (a bearer token is just a header, kept in memory instead of a header file). - Gaps: no `HTTP_PROXY`/`HTTPS_PROXY`/`NO_PROXY` support, HTTP/1.1 only, no automatic content decoding. ## Decisions 1. **One synchronous helper, one task per request.** A new `compiler/src/package/network.trb` holds the only `Task.runToEnd()` call site: it runs one `async` function that builds a `Client` *inside* the task (a `Client` is a `shared type` that stays with the task that made it), sends, reads the body with a limit and returns plain values. A fresh `Client` per request: a package resolution makes a handful of requests, and no connection or pool state has to survive between two scheduler runs. `fetch.trb` and `upload.trb` call only this helper. Nothing else in `compiler/src` calls `runToEnd()`, and no top level gets an `await()` (`main.trb` stays a synchronous top level, so `torb run`/`torb test` still start the VM scheduler). 2. **Redirects.** Fetches (index, `config.trb`, owner file, archive, `archive:` URL, mirrors alike) use `RedirectPolicy.AnyHost(5)`: no credential is ever sent on a fetch and every byte is checked (index signatures, tree hash), so a CDN hop is safe, and it keeps what `curl -L` did. `https` to `http` is never followed (stricter than `curl -L`). Requests that carry a token (`PUT` upload, `login`, `owner`, `yank`, `deprecate`, the trusted-publishing exchange, the CI identity-token `GET`) use `RedirectPolicy.Never`, as the old `curl` call had no `-L`: a 3xx is reported as a refusal (`<url> answered 301, redirecting to <Location>`, `printable`), so a token is never re-sent anywhere. 3. **Status mapping unchanged.** 404 and 410 of a fetch are "not found" (`Ok(false)`, the index case "the registry does not have the package"); any other non-2xx or an unreachable host is a `Fail` naming the URL and the status/cause in `printable` text. A request with a token returns the status and body as a `Reply` as before (a refusal is a `Reply`, not an error). 4. **Limits (numbers).** Head deadline `Client(timeout: Some(30.seconds()))` (curl had none; a hung registry should not hang `torb add`); fetch bodies `bytes(limit: 16 MiB)` (the archive cap is 10 MiB compressed); API replies `bytes(limit: 1 MiB)`; a non-UTF-8 API reply reads as empty text as before. 5. **Token handling.** The token travels as a header value in memory; no header file, answer file, `0700` upload directory or `0600` helper file is needed, so `uploadDirectory`, `writePrivate`, `RequestFiles`, `requestArguments` and the `work` parameters go away, `exchangedToken`'s `send` callback receives the JSON text instead of a file path, and `request(...)` keeps its `method, url, headers, body` shape for `cli/registry.trb`. The rule "token only over `https:` or loopback `http:`, never to a mirror" (`tokenOf`, `credentials.trb`) is unchanged. 6. **`TORB_FETCH` is removed**, as the acceptance criteria say (documented in `torb-install.md`, `torb-publish.md`; also set in `tools/registry/tests/commands.test.trb` and `owner-step-up.test.trb`, which lose it). `file:` registries, `--offline`, mirror order, signature and tree-hash checks are untouched (`fetchFirst` keeps its logic and only `fetchUrl` changes; the `download.part` file may stay). 7. **Known limitation, documented, not a blocker:** `std/http` does not read proxy variables, which `curl`/`wget` honoured implicitly. `docs/tooling/torb-install.md` states "the client does not use a proxy yet". A proxy follow-up is filed only when someone needs it. 8. **Link impact is accepted:** `torb` will link `std/http`, `std/tls`, `std/dns` and the vendored mbedTLS (compiled only where reached). No feature flag. The implementer reports the binary size and bootstrap time change from the forge CI run in the PR body; a red gate is fixed on the PR. ## Acceptance criteria (replace those in the issue body) - [ ] `network.trb` (helper above) is the only `runToEnd()` caller in `compiler/src`; `fetch.trb` downloads index files, `config.trb`, owner files and archives through it and writes them with `File.writeBytesAtomically`; 404/410 of an index is still "the registry does not have the package"; redirects as in decision 2 - [ ] `upload.trb` (`torb publish`, `login`, `owner`, `yank`, `deprecate`, trusted publishing) sends through the same helper with `Never` redirects; the token appears in no process argument and in no file - [ ] No `curl`/`wget`/`TORB_FETCH` in `compiler/src/package/`; a machine without them can `torb add`, `install`, `publish` (a test runs the `torb` binary as a subprocess against a loopback registry served by `std/http`, with `PATH` pointing at a directory that has neither tool, in the style of `tools/registry/tests/client.test.trb`) - [ ] Tests for the helper: 404 and 410 on fetch, a 500, a same-host and a cross-host redirect on fetch, a redirect on a token request is a refusal that sends the token nowhere, an `https`-to-`http` redirect is not followed, a body over the limit, an unreachable host names the URL. If `test` bodies turn out to run inside a task (so `runToEnd()` would panic), put these in `tests/conformance/` programs with a synchronous top level, as `task-run-to-end-http.trb` does - [ ] `--offline`, mirrors, signature and tree-hash checks behave unchanged; the existing suites (`compiler/tests/packages.test.trb`, `registry-commands.test.trb`, `tools/registry/tests/*`) stay green after their `curl`-specific parts (the fake-`curl` argument log, `requestArguments`, `uploadDirectory`) are replaced by the new tests - [ ] `torb run` and `torb test` still start the VM scheduler (no top-level task is introduced) - [ ] `RELEASE.md` 7.13 "How the client fetches" row, the module comments of `fetch.trb`/`upload.trb`/`store.trb` ("everything here is synchronous because ...") and `docs/tooling/torb-install.md`, `torb-add.md`, `torb-publish.md`, `torb-login.md` describe the new behaviour (including the proxy limitation); `skills/` regenerated - [ ] `torb upgrade` and the `Url` source of `torb template` (`compiler/src/cli/upgrade.trb`, `cli/template.trb`) stay on `curl`/`wget` unless the change is mechanical with the new helper; if left, their doc comments keep saying why - [ ] PR body reports the link-size and bootstrap-time change (decision 8) ## Seed and size - Seed impact: **none** (no new native, no name the compiler looks up by string; `runToEnd` is already in the published seed). If the bootstrap in CI fails because `std/http` at `main` uses a form the newest published seed cannot read, that is a new wait on a seed publication, not a design change. - Size: one PR, one Conventional Commit `feat(package): fetch and upload through std/http` (a second `docs(package): ...` commit is fine if the docs and `skills/` regeneration are easier to review apart). Not an epic. - Design record: `docs/design/RELEASE.md` 7.4, 7.5, 7.13; `docs/design/NETWORK.md` (client, redirects); `docs/standard-library/task.md` (`runToEnd`). ROADMAP: "The package manager" bullet.
Author
Collaborator

[loop:torbscript] refine: dependency #30 verified met (landed 7bfb4088; published seeds 4b3045d31334 and acf8510278d7 declare runToEnd), std/http covers HTTPS with platform trust, redirects with an origin policy, bearer headers and bodies of known length; scope, redirect/timeout/limit numbers, token handling and tests decided above, next: ready (implement). Tokens: ~110k (sonnet worker, estimate).

[loop:torbscript] refine: dependency #30 verified met (landed `7bfb4088`; published seeds `4b3045d31334` and `acf8510278d7` declare `runToEnd`), `std/http` covers HTTPS with platform trust, redirects with an origin policy, bearer headers and bodies of known length; scope, redirect/timeout/limit numbers, token handling and tests decided above, next: ready (implement). Tokens: ~110k (sonnet worker, estimate).
Author
Collaborator

[loop:torbscript] claim implement by session:loop-torbscript-20261029 until 2026-10-29T16:18Z

[loop:torbscript] claim implement by session:loop-torbscript-20261029 until 2026-10-29T16:18Z
Author
Collaborator

[loop:torbscript] implement: checkpoint 1: worktree loop/333-package-std-http from main @ 27fe87e3, surveyed fetch.trb, upload.trb, cli/registry.trb and the curl-specific tests; writing package/network.trb (the single runToEnd() helper) next, then fetch.trb, upload.trb, cli/registry.trb, tests and docs, next: implementing.

[loop:torbscript] implement: checkpoint 1: worktree `loop/333-package-std-http` from `main` @ 27fe87e3, surveyed `fetch.trb`, `upload.trb`, `cli/registry.trb` and the curl-specific tests; writing `package/network.trb` (the single `runToEnd()` helper) next, then `fetch.trb`, `upload.trb`, `cli/registry.trb`, tests and docs, next: implementing.
Author
Collaborator

Checkpoint 2 (implement): PR #579 is open (Closes #333), review requested from argus. The package manager now fetches and uploads through std/http (compiler/src/package/network.trb, the only runToEnd call); TORB_FETCH, the header and answer files and the upload directory are gone. Tests: packages.test.trb runs two loopback servers (126 passed in the VM), and the registry subprocess tests run torb with a PATH that holds no curl or wget. Waiting for the forge CI; the link size and bootstrap time will be reported in the PR.

**Checkpoint 2 (implement):** PR https://git.torb.dev/torbscript/language/pulls/579 is open (`Closes #333`), review requested from `argus`. The package manager now fetches and uploads through `std/http` (`compiler/src/package/network.trb`, the only `runToEnd` call); `TORB_FETCH`, the header and answer files and the upload directory are gone. Tests: `packages.test.trb` runs two loopback servers (126 passed in the VM), and the registry subprocess tests run `torb` with a `PATH` that holds no `curl` or `wget`. Waiting for the forge CI; the link size and bootstrap time will be reported in the PR.
Author
Collaborator

[loop:torbscript] claim fix by session:loop-torbscript-20261029 until 2026-10-29T17:26Z

[loop:torbscript] claim fix by session:loop-torbscript-20261029 until 2026-10-29T17:26Z
argus closed this issue 2026-10-29 22:16:56 +00:00
Sign in to join this conversation.
No labels
loop
ready
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
torbscript/language#333
No description provided.