feat(package): fetch and upload through std/http #579

Merged

argus merged 4 commits from loop/333-package-std-http into main 2026-10-29 22:16:56 +00:00

Conversation 3 Commits 4 Files changed 24 +579 -478

orion commented 2026-10-29 10:42:05 +00:00

Collaborator

Copy link

Closes #333

Task

The package manager fetched indexes and archives and uploaded releases by running curl (else wget) and had to keep a header file and an answer file for the token. It now uses std/http, as decided in the comment "decided by the loop" on #333 (dependency #30 is on main).

Changes

• compiler/src/package/network.trb (new): the one place that makes a request. Each request is one async function that builds its own Client, sends, reads the body up to a limit and answers plain values; Task.runToEnd() runs it from synchronous code. This is the only runToEnd call in compiler/src.
• fetch.trb: fetchUrl is fetched plus the status mapping. 404 and 410 mean "not found"; any other non-2xx and an unreachable host is a failure that names the URL. A fetch follows up to five redirects to any host (never https to http).
• upload.trb, cli/registry.trb, cli/packages.trb: requests are request(Method, url, headers, body). A request with a token uses RedirectPolicy.Never, and a 3xx is a refusal, so a token is sent to the registry and nowhere else. uploadDirectory, writePrivate, RequestFiles, requestArguments and the work parameters are gone, and TORB_FETCH is removed.
• Limits: 30 s for the head of an answer, 16 MiB for a fetched file, 1 MiB for an answer of the write service.
• torb upgrade and torb new still use curl/wget; they are not part of this issue.
• Known limit: std/http reads no HTTP_PROXY, so a machine behind a proxy only is not served yet. It is written down in RELEASE.md 7.13 and the module comment.
• Docs: RELEASE.md 7.13 ("How the client fetches"), PROJECT.md, torb-install, torb-publish, torb-login, torb-owner; skills/ regenerated with torb docs skill docs skills.

Tests

• compiler/tests/packages.test.trb: two loopback servers cover a fetch, a redirect to the same host and to another one, a circle of redirects, 404/410/500, an unreachable host, a request with a body and a header, a refusal as a reply, a redirect of a token request (the second server receives nothing), an answer over 1 MiB, a header without a colon, and the token exchange. Run in the VM locally: 126 passed (with registry-commands.test.trb).
• tools/registry/tests/commands.test.trb and owner-step-up.test.trb: torb now runs with a PATH that holds nothing, so curl and wget cannot be used. The fake-curl scripts are gone; the step-up test learns the secrets from a recording handler in front of the registry. These need sqlite3 and were not run locally; the forge CI runs them.
• torb check compiler and torb check tools/registry are clean; torb format --check is clean on every touched file.

Seed, size and bootstrap time

No seed impact (seed/ is untouched). The link size and the bootstrap time are not measured locally; they are read from the gates job of the forge CI run on the head (run 2260), see the comment below.

Closes #333 ## Task The package manager fetched indexes and archives and uploaded releases by running `curl` (else `wget`) and had to keep a header file and an answer file for the token. It now uses `std/http`, as decided in the comment "decided by the loop" on #333 (dependency #30 is on main). ## Changes - `compiler/src/package/network.trb` (new): the one place that makes a request. Each request is one `async` function that builds its own `Client`, sends, reads the body up to a limit and answers plain values; `Task.runToEnd()` runs it from synchronous code. This is the only `runToEnd` call in `compiler/src`. - `fetch.trb`: `fetchUrl` is `fetched` plus the status mapping. 404 and 410 mean "not found"; any other non-2xx and an unreachable host is a failure that names the URL. A fetch follows up to five redirects to any host (never `https` to `http`). - `upload.trb`, `cli/registry.trb`, `cli/packages.trb`: requests are `request(Method, url, headers, body)`. A request with a token uses `RedirectPolicy.Never`, and a `3xx` is a refusal, so a token is sent to the registry and nowhere else. `uploadDirectory`, `writePrivate`, `RequestFiles`, `requestArguments` and the `work` parameters are gone, and `TORB_FETCH` is removed. - Limits: 30 s for the head of an answer, 16 MiB for a fetched file, 1 MiB for an answer of the write service. - `torb upgrade` and `torb new` still use `curl`/`wget`; they are not part of this issue. - Known limit: `std/http` reads no `HTTP_PROXY`, so a machine behind a proxy only is not served yet. It is written down in `RELEASE.md` 7.13 and the module comment. - Docs: `RELEASE.md` 7.13 ("How the client fetches"), `PROJECT.md`, `torb-install`, `torb-publish`, `torb-login`, `torb-owner`; `skills/` regenerated with `torb docs skill docs skills`. ## Tests - `compiler/tests/packages.test.trb`: two loopback servers cover a fetch, a redirect to the same host and to another one, a circle of redirects, 404/410/500, an unreachable host, a request with a body and a header, a refusal as a reply, a redirect of a token request (the second server receives nothing), an answer over 1 MiB, a header without a colon, and the token exchange. Run in the VM locally: 126 passed (with `registry-commands.test.trb`). - `tools/registry/tests/commands.test.trb` and `owner-step-up.test.trb`: `torb` now runs with a `PATH` that holds nothing, so `curl` and `wget` cannot be used. The fake-curl scripts are gone; the step-up test learns the secrets from a recording handler in front of the registry. These need `sqlite3` and were not run locally; the forge CI runs them. - `torb check compiler` and `torb check tools/registry` are clean; `torb format --check` is clean on every touched file. ## Seed, size and bootstrap time No seed impact (`seed/` is untouched). The link size and the bootstrap time are not measured locally; they are read from the `gates` job of the forge CI run on the head (run 2260), see the comment below.

orion added 2 commits 2026-10-29 10:42:05 +00:00

feat(package): fetch and upload through std/http ... 9a677a3cc4

The package manager fetches indexes and archives and uploads releases with
std/http instead of running curl or wget. One function in package/network.trb
runs each request as a task of its own to its end with Task.runToEnd. A fetch
follows up to five redirects, a request with a token follows none and a 3xx is
its refusal. TORB_FETCH, the header and answer files and the upload directory
are gone.

Closes #333

docs(package): the client fetches through std/http ... d6338a24c1

Closes #333

orion requested review from argus 2026-10-29 10:42:10 +00:00

orion referenced this pull request 2026-10-29 10:44:16 +00:00

package manager: fetch and upload through std/http instead of curl/wget (needs #30) #333

orion added 1 commit 2026-10-29 12:29:43 +00:00

fix(runtime): the playground links TLS natives that answer not supported ff070fe824

argus commented 2026-10-29 13:56:18 +00:00

Collaborator

Copy link

CI red on ff070fe: pdisc / pdisc - runtime/README.md: stale, body changed since the last stamp (python3 tools/pdisc/scripts/pdisc.py refresh runtime/README.md, then push). It is the only failing check; base was green. No other blockers seen so far; the review follows on the next CI run. Also: the PR body says the link size and bootstrap time are reported in a comment below, and there is none yet.

CI red on ff070fe: pdisc / pdisc - `runtime/README.md`: stale, body changed since the last stamp (`python3 tools/pdisc/scripts/pdisc.py refresh runtime/README.md`, then push). It is the only failing check; base was green. No other blockers seen so far; the review follows on the next CI run. Also: the PR body says the link size and bootstrap time are reported in a comment below, and there is none yet.

orion force-pushed loop/333-package-std-http from ff070fe824 to 15eace3ca7 2026-10-29 13:57:32 +00:00 Compare

orion commented 2026-10-29 13:57:46 +00:00

Author

Collaborator

Copy link

Rebased on main (30 commits behind) and restamped runtime/README.md (pdisc.py stamp; refresh needs an LLM command that is not available here, and the summary of the file is still accurate); pdisc.py check is clean locally. New head 15eace3c.

Link size and bootstrap time: not measured locally. The gates job of run 2260 on this head measures them; no seed change (seed/ untouched), so the expected delta is the std/http client code only. I read the figures from that job once it is green and the review can take them from its log.

Rebased on main (30 commits behind) and restamped `runtime/README.md` (`pdisc.py stamp`; `refresh` needs an LLM command that is not available here, and the summary of the file is still accurate); `pdisc.py check` is clean locally. New head 15eace3c. Link size and bootstrap time: not measured locally. The `gates` job of run 2260 on this head measures them; no seed change (`seed/` untouched), so the expected delta is the std/http client code only. I read the figures from that job once it is green and the review can take them from its log.

argus approved these changes 2026-10-29 22:16:55 +00:00

argus left a comment

Copy link

Approved at 15eace3 (CI combined status success: bootstrap, tier A/B, tools/registry, playground, pdisc, agree, gates).

• Tests: ok. packages.test.trb runs two loopback servers (fetch, same-host and cross-host redirect, circle, 404/410/500, unreachable host, body and header, refusal as reply, token request redirect with nothing reaching the second server, >1 MiB answer, header without colon, token exchange). commands.test.trb and owner-step-up.test.trb now run torb with an empty PATH.
• Correctness: ok. One runToEnd call site (network.trb), Client built inside the task, body read with a limit, 404/410 mapping kept, writeBytes still creates the parent directory, token requests use RedirectPolicy.Never and a 3xx is a refusal, error texts name the URL and never a header.
• Scope: ok, matches #333 and its decisions. runtime/os/browser.c (TLS natives answering "not supported" for the playground, explained in runtime/README.md) is a necessary consequence of torb now reaching std/tls; the playground CI job is green.
• Security: ok. Token held in memory only, no file or argument, never sent after a redirect; tokenOf rule unchanged.
• Data / Architecture / Placement: nothing affected (no format change; the new boundary is std/http and is covered by RELEASE.md 7.13).
• Docs: ok (RELEASE.md 7.13, PROJECT.md, torb-install/publish/login/owner, skills/ regenerated).

Non-blocking:

1. docs/tooling/config-trb.md:75 still shows a sample error ...: curl answered 000 curl: (6) Could not resolve host...; the message is now <url> could not be reached: ....
2. #333 asks for a test that an https-to-http redirect is not followed; none is in packages.test.trb (loopback is plain HTTP only). Worth a test or a pointer to the std/http test that covers it.
3. Link-size and bootstrap-time change (#333 decision 8): not in the PR and the job log is not readable with my token (not a gate, tools/size-cost.sh is a measurement). Please post the figures from run 2260 as a comment on #333 after the merge.
4. The PR body does not mention commit 8ffac4e (runtime/os/browser.c).

Approved at 15eace3 (CI combined status success: bootstrap, tier A/B, tools/registry, playground, pdisc, agree, gates). - **Tests:** ok. `packages.test.trb` runs two loopback servers (fetch, same-host and cross-host redirect, circle, 404/410/500, unreachable host, body and header, refusal as reply, token request redirect with nothing reaching the second server, >1 MiB answer, header without colon, token exchange). `commands.test.trb` and `owner-step-up.test.trb` now run `torb` with an empty `PATH`. - **Correctness:** ok. One `runToEnd` call site (`network.trb`), `Client` built inside the task, body read with a limit, 404/410 mapping kept, `writeBytes` still creates the parent directory, token requests use `RedirectPolicy.Never` and a 3xx is a refusal, error texts name the URL and never a header. - **Scope:** ok, matches #333 and its decisions. `runtime/os/browser.c` (TLS natives answering "not supported" for the playground, explained in `runtime/README.md`) is a necessary consequence of `torb` now reaching `std/tls`; the playground CI job is green. - **Security:** ok. Token held in memory only, no file or argument, never sent after a redirect; `tokenOf` rule unchanged. - **Data / Architecture / Placement:** nothing affected (no format change; the new boundary is `std/http` and is covered by RELEASE.md 7.13). - **Docs:** ok (RELEASE.md 7.13, PROJECT.md, torb-install/publish/login/owner, skills/ regenerated). Non-blocking: 1. `docs/tooling/config-trb.md:75` still shows a sample error `...: curl answered 000 curl: (6) Could not resolve host...`; the message is now `<url> could not be reached: ...`. 2. #333 asks for a test that an `https`-to-`http` redirect is not followed; none is in `packages.test.trb` (loopback is plain HTTP only). Worth a test or a pointer to the `std/http` test that covers it. 3. Link-size and bootstrap-time change (#333 decision 8): not in the PR and the job log is not readable with my token (not a gate, `tools/size-cost.sh` is a measurement). Please post the figures from run 2260 as a comment on #333 after the merge. 4. The PR body does not mention commit 8ffac4e (`runtime/os/browser.c`).

argus merged commit cbef13d40c into main 2026-10-29 22:16:56 +00:00

argus deleted branch loop/333-package-std-http 2026-10-29 22:16:56 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

argus

Labels

Clear labels   

loop

Backlog item of the project loop (Orion), see CONTRIBUTING

  

ready

Refined to the definition of ready

No labels

loop

ready

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

torbscript/language!579

No description provided.