package manager: resolve git: sources #322

Closed
opened 2026-10-12 08:24:42 +00:00 by orion · 7 comments
Collaborator

Context

Split from #27. A git: source of a manifest is refused by the resolver ("is not resolved yet"). #27 resolves archive: sources and adds the lock rows, resolver and cache plumbing; this issue adds git:.

Source

Acceptance criteria

  • A git: source is fetched at its revision: (tag, branch or commit) with the git executable; the resolved commit and the tree hash of the checkout (without .git) are recorded in the lock as package "<name>", git: "<Uri>", revision: "<commit>", hash: "sha256:..."
  • A locked source is fetched at the locked commit and checked against the locked hash; a changed hash is an error naming both
  • The tree is installed in the content-addressed cache under its tree hash; with --offline or a warm cache, no git or network call is made
  • The package name from the fetched project.trb must equal the source name
  • Tests with a local repository (file: URL, created by the test), including a tag and a branch revision
  • A missing git executable gives an error naming it; docs (project-lock-trb, project-trb, PROJECT.md "As built") updated; skills/ regenerated

Seed impact

None (ordinary compiler code, no syntax change, native or driver flag).

Dependencies

  • Depends on #27 (lock rows, resolver and cache plumbing).
## Context Split from #27. A `git:` source of a manifest is refused by the resolver ("is not resolved yet"). #27 resolves `archive:` sources and adds the lock rows, resolver and cache plumbing; this issue adds `git:`. ## Source - [PROJECT.md sections 7 and 11](https://git.torb.dev/torbscript/language/src/branch/main/docs/design/PROJECT.md) (`revision:` required; lock pins the resolved commit and the tree hash) - [ROADMAP.md, the package manager](https://git.torb.dev/torbscript/language/src/branch/main/docs/ROADMAP.md) ## Acceptance criteria - [ ] A `git:` source is fetched at its `revision:` (tag, branch or commit) with the `git` executable; the resolved commit and the tree hash of the checkout (without `.git`) are recorded in the lock as `package "<name>", git: "<Uri>", revision: "<commit>", hash: "sha256:..."` - [ ] A locked source is fetched at the locked commit and checked against the locked hash; a changed hash is an error naming both - [ ] The tree is installed in the content-addressed cache under its tree hash; with `--offline` or a warm cache, no git or network call is made - [ ] The package name from the fetched `project.trb` must equal the `source` name - [ ] Tests with a local repository (`file:` URL, created by the test), including a tag and a branch revision - [ ] A missing `git` executable gives an error naming it; docs (project-lock-trb, project-trb, PROJECT.md "As built") updated; `skills/` regenerated ## Seed impact None (ordinary compiler code, no syntax change, native or driver flag). ## Dependencies - Depends on #27 (lock rows, resolver and cache plumbing).
Author
Collaborator

[loop:torbscript] intake: split from #27, in scope (author orion, label loop). Design decisions: git is run as an executable like curl/wget (no libgit), the lock pins commit plus tree hash, cache is content-addressed. Refine at the stage where #27 has landed the lock rows, next: waiting (dependency #27)

[loop:torbscript] intake: split from #27, in scope (author orion, label loop). Design decisions: `git` is run as an executable like curl/wget (no libgit), the lock pins commit plus tree hash, cache is content-addressed. Refine at the stage where #27 has landed the lock rows, next: waiting (dependency #27)
Author
Collaborator

[loop:torbscript] claim refine by session:loop-torbscript-20261012 until 2026-10-12T14:08Z

[loop:torbscript] claim refine by session:loop-torbscript-20261012 until 2026-10-12T14:08Z
Author
Collaborator

[loop:torbscript] decided by the loop: refinement of the git: resolver, against the landed #27 (unpackedSource, fetchSourceArchive, LockedPackage.archive, SourceBinding.pin/lockedLocation; the refusal sits in compiler/src/cli/packages.trb at the "git" kind of the source match).

  • Lock row: package "<name>", git: "<Uri>", revision: "<commit>", hash: "sha256:..." as in the acceptance criteria, plus ref: "<revision as written>" between git: and revision: only when the written revision is not the full commit id (tag or branch). Why: the source line pins a movable name, the lock pins the commit; without ref: the build could not tell that a source line changed from tag v1 to v2, which is the mismatch check #27 already does for archive:. A full 40-hex revision gets no ref:. New field LockedPackage.git (+ ref, commit), isGit(), a described() case git <url> <commit>, and the same mismatch rules as archive: (URL, ref, or hash differs from the source line).
  • Fetch: run git as an executable through Process.runBlocking (no libgit), in a scratch directory under the cache: git init, git fetch --depth 1 <url> <rev> (tag, branch, or commit), falling back to a full git fetch when the server refuses a shallow fetch of that name; git rev-parse FETCH_HEAD^{commit} gives the commit; git checkout --detach. A locked source fetches the locked commit id instead of the written revision. git environment: GIT_TERMINAL_PROMPT=0 (never prompt), no submodules (a checkout with a .gitmodules is fetched without them and documented as such). file: URLs are relative to the workspace root like archive:.
  • Tree hash: the checkout without .git is read into the same TarEntry list as an unpacked archive and goes through the registry's safety checks and treeHash of ./archive, so a git: tree and an archive: tree of the same files hash alike. Symlinks and the other things an archive refuses are refused here too. Installed with install(cache, hash, entries) under the tree hash; isInstalled first, so a warm cache and --offline make no git or network call.
  • Offline without a lock row: the commit is unknown, so it is an error naming the package (the lock row or the cache is what --offline reads).
  • Name check: the fetched project.trb must name the source package, as for archive:; its dependencies join the resolution like a path: package's.
  • Other commands: vendor leaves a git: package in place and publish refuses a package that depends on one, both exactly as for archive:; torb update re-resolves ref to the current commit.
  • Missing git: if the executable cannot be started: "fetching <url> needs git, and it could not be started"; TORB_GIT may name another executable, like TORB_FETCH does for curl/wget.
  • Seed impact: none (ordinary compiler code in compiler/src/, no syntax, native or driver flag). Size: one PR, feat(package): resolve git: sources plus docs(tooling): git: sources in the lock, similar in size to #27 (about 15 files).

Acceptance criteria (refined, all testable offline with local repositories)

  • compiler/tests/packages.test.trb: the test creates a repository in a temp directory with the git executable (git init, commit of a project.trb and one source file, git tag v1, a branch next with a second commit) and a file: URL to it. Resolving with revision: "v1" gives a lock row with ref: "v1", the commit of the tag, and the tree hash; revision: "next" gives the branch's commit; a full commit id gives a row without ref:.
  • The tree hash equals the treeHash of the same files packed as an archive: (the .git directory never counts), and the package is installed in the cache under it.
  • A locked source is fetched at the locked commit even when the tag or branch moved afterwards (test moves the branch); a tree whose hash differs from the lock is an error naming the locked and the actual hash.
  • Warm cache and --offline: after removing the repository, the build still resolves from the cache and starts no git (test sets TORB_GIT to a program that fails). --offline with no lock row and a cold cache is an error naming the package.
  • A fetched project.trb that names another package than the source line is an error naming both.
  • A source line whose revision: or URL changed against the lock is reported as a mismatch by the build, lock --check and install, as for archive:; vendor leaves the package, publish refuses it.
  • TORB_GIT naming a program that does not exist gives an error that names git.
  • tools/packages.sh transcript (tests/packages/transcript.expected): a git-app fixture whose repository the script creates with git (fixed author/dates so the commit id is stable) covers lock, run, lock --check and a mismatch; no network.
  • Docs updated: docs/tooling/project-lock-trb.md, docs/tooling/project-trb.md, docs/design/PROJECT.md section 7 and 11 "As built" (replace "git: rows are not written yet" and "refused by the resolver"), docs/how-to/add-a-dependency.md; skills/ regenerated.

Design: PROJECT.md sections 7 and 11 (revision: required; lock pins commit and tree hash); ROADMAP "the package manager". Depends on #27, which is on main.

[loop:torbscript] decided by the loop: refinement of the `git:` resolver, against the landed #27 (`unpackedSource`, `fetchSourceArchive`, `LockedPackage.archive`, `SourceBinding.pin/lockedLocation`; the refusal sits in `compiler/src/cli/packages.trb` at the `"git"` kind of the source match). - **Lock row**: `package "<name>", git: "<Uri>", revision: "<commit>", hash: "sha256:..."` as in the acceptance criteria, plus `ref: "<revision as written>"` between `git:` and `revision:` only when the written revision is not the full commit id (tag or branch). Why: the `source` line pins a movable name, the lock pins the commit; without `ref:` the build could not tell that a `source` line changed from tag `v1` to `v2`, which is the mismatch check #27 already does for `archive:`. A full 40-hex revision gets no `ref:`. New field `LockedPackage.git` (+ `ref`, `commit`), `isGit()`, a `described()` case `git <url> <commit>`, and the same mismatch rules as `archive:` (URL, ref, or hash differs from the `source` line). - **Fetch**: run `git` as an executable through `Process.runBlocking` (no libgit), in a scratch directory under the cache: `git init`, `git fetch --depth 1 <url> <rev>` (tag, branch, or commit), falling back to a full `git fetch` when the server refuses a shallow fetch of that name; `git rev-parse FETCH_HEAD^{commit}` gives the commit; `git checkout --detach`. A locked source fetches the locked commit id instead of the written revision. `git` environment: `GIT_TERMINAL_PROMPT=0` (never prompt), no submodules (a checkout with a `.gitmodules` is fetched without them and documented as such). `file:` URLs are relative to the workspace root like `archive:`. - **Tree hash**: the checkout without `.git` is read into the same `TarEntry` list as an unpacked archive and goes through the registry's safety checks and `treeHash` of `./archive`, so a `git:` tree and an `archive:` tree of the same files hash alike. Symlinks and the other things an archive refuses are refused here too. Installed with `install(cache, hash, entries)` under the tree hash; `isInstalled` first, so a warm cache and `--offline` make no git or network call. - **Offline without a lock row**: the commit is unknown, so it is an error naming the package (the lock row or the cache is what `--offline` reads). - **Name check**: the fetched `project.trb` must name the `source` package, as for `archive:`; its dependencies join the resolution like a `path:` package's. - **Other commands**: `vendor` leaves a `git:` package in place and `publish` refuses a package that depends on one, both exactly as for `archive:`; `torb update` re-resolves `ref` to the current commit. - **Missing `git`**: if the executable cannot be started: "fetching `<url>` needs `git`, and it could not be started"; `TORB_GIT` may name another executable, like `TORB_FETCH` does for curl/wget. - **Seed impact**: none (ordinary compiler code in `compiler/src/`, no syntax, native or driver flag). Size: one PR, `feat(package): resolve git: sources` plus `docs(tooling): git: sources in the lock`, similar in size to #27 (about 15 files). ## Acceptance criteria (refined, all testable offline with local repositories) - [ ] `compiler/tests/packages.test.trb`: the test creates a repository in a temp directory with the `git` executable (`git init`, commit of a `project.trb` and one source file, `git tag v1`, a branch `next` with a second commit) and a `file:` URL to it. Resolving with `revision: "v1"` gives a lock row with `ref: "v1"`, the commit of the tag, and the tree hash; `revision: "next"` gives the branch's commit; a full commit id gives a row without `ref:`. - [ ] The tree hash equals the `treeHash` of the same files packed as an `archive:` (the `.git` directory never counts), and the package is installed in the cache under it. - [ ] A locked source is fetched at the locked commit even when the tag or branch moved afterwards (test moves the branch); a tree whose hash differs from the lock is an error naming the locked and the actual hash. - [ ] Warm cache and `--offline`: after removing the repository, the build still resolves from the cache and starts no `git` (test sets `TORB_GIT` to a program that fails). `--offline` with no lock row and a cold cache is an error naming the package. - [ ] A fetched `project.trb` that names another package than the `source` line is an error naming both. - [ ] A `source` line whose `revision:` or URL changed against the lock is reported as a mismatch by the build, `lock --check` and `install`, as for `archive:`; `vendor` leaves the package, `publish` refuses it. - [ ] `TORB_GIT` naming a program that does not exist gives an error that names `git`. - [ ] `tools/packages.sh` transcript (`tests/packages/transcript.expected`): a `git-app` fixture whose repository the script creates with `git` (fixed author/dates so the commit id is stable) covers lock, run, `lock --check` and a mismatch; no network. - [ ] Docs updated: `docs/tooling/project-lock-trb.md`, `docs/tooling/project-trb.md`, `docs/design/PROJECT.md` section 7 and 11 "As built" (replace "`git:` rows are not written yet" and "refused by the resolver"), `docs/how-to/add-a-dependency.md`; `skills/` regenerated. Design: PROJECT.md sections 7 and 11 (`revision:` required; lock pins commit and tree hash); ROADMAP "the package manager". Depends on #27, which is on `main`.
Author
Collaborator

[loop:torbscript] refine: done, decisions and refined acceptance criteria recorded above, label ready set, next: implement (opus/high: compiler/src; about 130k tokens used by this worker)

[loop:torbscript] refine: done, decisions and refined acceptance criteria recorded above, label ready set, next: implement (opus/high: compiler/src; about 130k tokens used by this worker)
Author
Collaborator

[loop:torbscript] claim implement by session:loop-torbscript-20261012 until 2026-10-12T18:10Z

[loop:torbscript] claim implement by session:loop-torbscript-20261012 until 2026-10-12T18:10Z
Author
Collaborator

[loop:torbscript] implement: PR #330 open, forge CI running; the worker was terminated by a gateway restart after opening it. next: waiting (CI running on PR #330)

[loop:torbscript] implement: PR #330 open, forge CI running; the worker was terminated by a gateway restart after opening it. next: waiting (CI running on PR #330)
argus closed this issue 2026-10-12 15:45:14 +00:00
Author
Collaborator

[loop:torbscript] release implement: PR #330 landed on main (merged by the merge gate); the follow-up worker died of a session limit but its work was finished by a later worker. Nothing left on this issue.

[loop:torbscript] release implement: PR #330 landed on main (merged by the merge gate); the follow-up worker died of a session limit but its work was finished by a later worker. Nothing left on this issue.
Sign in to join this conversation.
No labels
loop
ready
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
torbscript/language#322
No description provided.