package manager: fetch and upload through std/http instead of curl/wget (needs #30) #333
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Context
Split from #30. The package manager fetches and uploads through
curl, elsewget(compiler/src/package/fetch.trb,compiler/src/package/upload.trb,RELEASE.mdsection 7.13 "How the client fetches") becausetorbcould not wait for a task. #30 adds a synchronous wait (Task.runToEnd()); this issue moves the client ontostd/http.Depends on
runToEnd()once the seed knows it;CLAUDE.md"Seed and breaking changes", two commits). The seed refresh is the owner's, so the loop treats this issue as waiting until the published seed contains the native.Acceptance criteria
fetch.trbdownloads an index and an archive withstd/http'sClient(redirects followed within the registry's origin policy, HTTPS with the platform's trust) and writes the bytes withFile.writeBytesAtomically; a 404 of an index is still "the registry does not have the package"upload.trb(torb publish,login,owner,yank,deprecaterequests) usesstd/httpthe same wayTORB_FETCHand everycurl/wgetspawn incompiler/src/package/are gone; a machine withoutcurlandwgetcantorb add,installandpublish(a test runs the client against a local registry served bystd/httpwithPATHemptied of both tools)--offline, mirrors, signature and tree-hash checks behave unchanged (existing tests stay green)torb runandtorb teststill start the VM scheduler (no top-level task is introduced)RELEASE.mdsection 7.13 table row, the docs-comments offetch.trb/upload.trb/store.trbanddocs/tooling/torb-add.mddescribe the new behaviour;skills/regeneratedtorb upgrade(compiler/src/cli/upgrade.trb) is out of scope here unless the change is mechanical; if left oncurl, its doc comment keeps saying whyDesign record
docs/design/RELEASE.mdsections 7.4, 7.5, 7.13;docs/design/NETWORK.md(client). Seed impact: none beyond the dependency above (no new native, no name the compiler looks up by string).Size
One PR, one Conventional Commit (
feat(package): fetch and upload through std/http).std/http#30std/http#30[loop:torbscript] refine: split from #30, not ready, next: waiting (dependency #30 merged and seed refreshed)
[loop:torbscript] claim refine by session:loop-torbscript-20261029 until 2026-10-29T10:20Z
[loop:torbscript] decided by the loop: the dependency is met on
main, and the shape of the migration is as follows (verified againstorigin/main@4b59fae0).Dependency check
7bfb4088(feat(task): run a task to its end from synchronous code, 2026-10-12):native fn runToEnd(): Result<Value, Cancelled>is declared instd/task/src/lib.trb, documented indocs/standard-library/task.md, andtests/conformance/task-run-to-end-http.trbruns astd/httpserver and client from a top level withoutawait().seeds.txtlists4b3045d31334(2026-10-29) andacf8510278d7first; both have7bfb4088as an ancestor and declarerunToEndinstd/task/src/lib.trb. CI bootstraps from the newest published seed, so the two-commit rule ofCLAUDE.mdis satisfied and no further seed refresh is needed. The "waiting on a seed refresh" condition is cleared.RELEASE.md7.13 gives forcurl(name resolution crashing on Windows before Winsock is loaded) was fixed on 2026-09-27 and checked on 2026-09-28; it is no reason againststd/httpany more.What
std/httpoffers (checked instd/http/src/client.trb,pool.trb,message.trb)Client(maximumConnections, timeout, redirects, tls, resolver); methodsget,post,send(Method.Put | Delete | Post ...);Body.from(Bytes)sends a knownContent-Length;response.body.bytes(limit:)collects a body up to a limit (default 16 MiB);Headers.set.httpsover mbedTLS with the platform's trust (crypt32 on Windows,$SSL_CERT_FILEor the system bundle on POSIX), the URL's host as the certificate name.RedirectPolicy.Never | SameHost(limit) | AnyHost(limit);httpstohttpis never followed;AnyHoststripsAuthorization,Cookie,Proxy-Authorizationwhen the host changes; a redirect that would have to resend a streamed body is returned as the response.HTTP_PROXY/HTTPS_PROXY/NO_PROXYsupport, HTTP/1.1 only, no automatic content decoding.Decisions
compiler/src/package/network.trbholds the onlyTask.runToEnd()call site: it runs oneasyncfunction that builds aClientinside the task (aClientis ashared typethat stays with the task that made it), sends, reads the body with a limit and returns plain values. A freshClientper request: a package resolution makes a handful of requests, and no connection or pool state has to survive between two scheduler runs.fetch.trbandupload.trbcall only this helper. Nothing else incompiler/srccallsrunToEnd(), and no top level gets anawait()(main.trbstays a synchronous top level, sotorb run/torb teststill start the VM scheduler).config.trb, owner file, archive,archive:URL, mirrors alike) useRedirectPolicy.AnyHost(5): no credential is ever sent on a fetch and every byte is checked (index signatures, tree hash), so a CDN hop is safe, and it keeps whatcurl -Ldid.httpstohttpis never followed (stricter thancurl -L). Requests that carry a token (PUTupload,login,owner,yank,deprecate, the trusted-publishing exchange, the CI identity-tokenGET) useRedirectPolicy.Never, as the oldcurlcall had no-L: a 3xx is reported as a refusal (<url> answered 301, redirecting to <Location>,printable), so a token is never re-sent anywhere.Ok(false), the index case "the registry does not have the package"); any other non-2xx or an unreachable host is aFailnaming the URL and the status/cause inprintabletext. A request with a token returns the status and body as aReplyas before (a refusal is aReply, not an error).Client(timeout: Some(30.seconds()))(curl had none; a hung registry should not hangtorb add); fetch bodiesbytes(limit: 16 MiB)(the archive cap is 10 MiB compressed); API repliesbytes(limit: 1 MiB); a non-UTF-8 API reply reads as empty text as before.0700upload directory or0600helper file is needed, souploadDirectory,writePrivate,RequestFiles,requestArgumentsand theworkparameters go away,exchangedToken'ssendcallback receives the JSON text instead of a file path, andrequest(...)keeps itsmethod, url, headers, bodyshape forcli/registry.trb. The rule "token only overhttps:or loopbackhttp:, never to a mirror" (tokenOf,credentials.trb) is unchanged.TORB_FETCHis removed, as the acceptance criteria say (documented intorb-install.md,torb-publish.md; also set intools/registry/tests/commands.test.trbandowner-step-up.test.trb, which lose it).file:registries,--offline, mirror order, signature and tree-hash checks are untouched (fetchFirstkeeps its logic and onlyfetchUrlchanges; thedownload.partfile may stay).std/httpdoes not read proxy variables, whichcurl/wgethonoured implicitly.docs/tooling/torb-install.mdstates "the client does not use a proxy yet". A proxy follow-up is filed only when someone needs it.torbwill linkstd/http,std/tls,std/dnsand the vendored mbedTLS (compiled only where reached). No feature flag. The implementer reports the binary size and bootstrap time change from the forge CI run in the PR body; a red gate is fixed on the PR.Acceptance criteria (replace those in the issue body)
network.trb(helper above) is the onlyrunToEnd()caller incompiler/src;fetch.trbdownloads index files,config.trb, owner files and archives through it and writes them withFile.writeBytesAtomically; 404/410 of an index is still "the registry does not have the package"; redirects as in decision 2upload.trb(torb publish,login,owner,yank,deprecate, trusted publishing) sends through the same helper withNeverredirects; the token appears in no process argument and in no filecurl/wget/TORB_FETCHincompiler/src/package/; a machine without them cantorb add,install,publish(a test runs thetorbbinary as a subprocess against a loopback registry served bystd/http, withPATHpointing at a directory that has neither tool, in the style oftools/registry/tests/client.test.trb)https-to-httpredirect is not followed, a body over the limit, an unreachable host names the URL. Iftestbodies turn out to run inside a task (sorunToEnd()would panic), put these intests/conformance/programs with a synchronous top level, astask-run-to-end-http.trbdoes--offline, mirrors, signature and tree-hash checks behave unchanged; the existing suites (compiler/tests/packages.test.trb,registry-commands.test.trb,tools/registry/tests/*) stay green after theircurl-specific parts (the fake-curlargument log,requestArguments,uploadDirectory) are replaced by the new teststorb runandtorb teststill start the VM scheduler (no top-level task is introduced)RELEASE.md7.13 "How the client fetches" row, the module comments offetch.trb/upload.trb/store.trb("everything here is synchronous because ...") anddocs/tooling/torb-install.md,torb-add.md,torb-publish.md,torb-login.mddescribe the new behaviour (including the proxy limitation);skills/regeneratedtorb upgradeand theUrlsource oftorb template(compiler/src/cli/upgrade.trb,cli/template.trb) stay oncurl/wgetunless the change is mechanical with the new helper; if left, their doc comments keep saying whySeed and size
runToEndis already in the published seed). If the bootstrap in CI fails becausestd/httpatmainuses a form the newest published seed cannot read, that is a new wait on a seed publication, not a design change.feat(package): fetch and upload through std/http(a seconddocs(package): ...commit is fine if the docs andskills/regeneration are easier to review apart). Not an epic.docs/design/RELEASE.md7.4, 7.5, 7.13;docs/design/NETWORK.md(client, redirects);docs/standard-library/task.md(runToEnd). ROADMAP: "The package manager" bullet.[loop:torbscript] refine: dependency #30 verified met (landed
7bfb4088; published seeds4b3045d31334andacf8510278d7declarerunToEnd),std/httpcovers HTTPS with platform trust, redirects with an origin policy, bearer headers and bodies of known length; scope, redirect/timeout/limit numbers, token handling and tests decided above, next: ready (implement). Tokens: ~110k (sonnet worker, estimate).[loop:torbscript] claim implement by session:loop-torbscript-20261029 until 2026-10-29T16:18Z
[loop:torbscript] implement: checkpoint 1: worktree
loop/333-package-std-httpfrommain@27fe87e3, surveyedfetch.trb,upload.trb,cli/registry.trband the curl-specific tests; writingpackage/network.trb(the singlerunToEnd()helper) next, thenfetch.trb,upload.trb,cli/registry.trb, tests and docs, next: implementing.Checkpoint 2 (implement): PR #579 is open (
Closes #333), review requested fromargus. The package manager now fetches and uploads throughstd/http(compiler/src/package/network.trb, the onlyrunToEndcall);TORB_FETCH, the header and answer files and the upload directory are gone. Tests:packages.test.trbruns two loopback servers (126 passed in the VM), and the registry subprocess tests runtorbwith aPATHthat holds nocurlorwget. Waiting for the forge CI; the link size and bootstrap time will be reported in the PR.[loop:torbscript] claim fix by session:loop-torbscript-20261029 until 2026-10-29T17:26Z