registry: search

orion commented

2026-10-05 21:07:10 +00:00

Collaborator

Goal

A visitor or a client can find packages in the registry by text: GET /api/1/search answers a ranked list over owner, name, description and keywords of what is published. The search page and a client command are not part of this issue (see "Not in this issue").

Source

ROADMAP.md, the package manager (open work: "the registry's sign-in, documentation worker, search and owners")
RELEASE.md section 7.9 "Search and ranking" (covers owner, name, description and keywords; keywords is added, not static, in the lock's settings, required by nothing; ranking after text relevance by dependents, then the registry's package checks)
RELEASE.md section 7.4 (the API carries search; "Description, keywords, the README and the documentation are on the API and the page", the signed index keeps only what resolution needs) and 7.11 ("Not built yet: ... search")
tools/registry/README.md, "Not built yet"

Findings from refine

The registry stores neither description nor keywords: releases has only package, version, hash, publisher, yank state; the index record is signed and resolution-only, and stays exactly as it is. The description is in the archive's project.lock.trb settings block, which checkedUpload already reads on every publish.
keywords is not in the vocabulary yet (std/project Project, Manifest, the lock's settings); every other setting 7.9 needs exists.
RELEASE.md does not ask for torb search (7.13 lists add, remove, update, install, publish; 7.13 "Not built yet" names yank, owner, login; nothing names search). The acceptance bullet "torb search ... if RELEASE.md asks for it" therefore resolves to: not built here.
Neither the capability summary, the documentation worker nor the "checks" of 7.9 (documented declarations, lowers on every target, README, license) exist as registry data yet (#51, #17/#237), so ranking on checks cannot be built now.

Design (decided by the loop, reasoning in the comment below)

Vocabulary: keywords "http", "client" in project.trb, variadic like authors: a variadic method of Project in std/project, Manifest.keywords, read by manifestOf and written by settingsLines into the lock's settings, listed in unreadWords (never evaluated by itself). publishProblems refuses at publish: more than 5 keywords, a keyword that is not 2 to 24 characters of lowercase a-z, 0-9, - starting with a letter or digit, or a duplicate. Nothing else changes for a package without keywords.
Data: a derived table search (package, owner, name, version, description, keywords, capabilities, published) holding one row per package that has at least one non-yanked release, plus search_dependencies (package, dependency) for the runtime dependencies of that same release. The row describes the latest non-yanked release (highest stable version; the highest pre-release only when no stable one is published). It is rewritten in the same statement batch as the release row on publish and on yank/unyank, under the existing WriteLock; a package whose releases are all yanked has no row. The index is the truth and is not touched; the table is derived and can be rebuilt.
Backfill: at start, every package that has releases but no search row is filled from the archive of its latest non-yanked release (unpacked and read the way checkedUpload does); a package that fails is logged and skipped, never fatal.
Matching and ranking: the query is split on whitespace, lower-cased, at most 8 terms of at most 64 characters; a package must match every term (AND). Per term the best of: exact name 16, name prefix 8, name contains 4, keyword equal 4, owner equal 2, description contains 1 (substring, instr over lower-cased text); the sum is the relevance. Order: relevance, then dependents (distinct packages whose latest release lists it as a runtime dependency), then newest publish, then package name. A term with / is matched against owner/name. Downloads are not ranked on (7.9).
API: GET /api/1/search?q=<text>&limit=<1..50, default 20>&offset=<default 0>, public, no token. Answer 200: {"query": "...", "total": N, "limit": N, "offset": N, "packages": [{"package": "acme/http", "owner": "acme", "name": "http", "version": "1.2.4", "description": "...", "keywords": ["http"], "capabilities": ["network"], "dependents": 3, "published": 1790000000}]} (integers are seconds since 1970 like the other calls). An empty q or one without a term, a bad limit or offset: 400 with the usual {"error", "problems"}. Cache-Control: public, max-age=60.
Why a scan and not FTS5: the registry reaches SQLite through the sqlite3 shell, whose build options are the container's and the CI's business; an instr scan over the derived table needs none. Measured with SQLite 3.40 on 20 000 synthetic packages (12 words of description, 4 keywords each): 18 ms per query including the client; the registry will hold orders of magnitude fewer for a long time. Revisit only when a measurement shows more than 100 ms.

Acceptance criteria

keywords exists in std/project, Manifest, the manifest reader and the lock's settings (round trip through torb lock and torb lock --check); torb publish --dry-run and the registry refuse the keyword cases above with one sentence each; a lock without keywords reads and writes exactly as before. docs/tooling/project-trb.md and the settings table of docs/design/PROJECT.md list it; skills/ regenerated.
tools/registry migrates an existing database in place (new tables created, existing accounts, tokens and releases untouched) and backfills the search rows at start.
A publish writes the package's search row from the lock's settings of the archive; a newer version replaces it; a yank of the latest version moves the row to the next newest non-yanked one, a yank of the only version removes it, an unyank restores it; the signed index and the archives are byte-identical to what they were before this change for the same inputs.
GET /api/1/search matches owner, name, description and keywords, requires every term, orders by the rules above (tests with fixtures that pin each tier: exact name above prefix above contains above keyword above description; equal relevance ordered by dependents), honours limit and offset, reports total, returns an empty packages list for no match, and refuses the malformed requests with 400.
A term that is SQL-hostile (', %, _, ;, control characters, 64+ characters) is matched literally or refused, never executed (test).
tests/server.test.trb (or a new test file) covers each case once as it works and once as it is refused; torb test tools/registry/tests green natively and in the VM; compiler tests for the keywords round trip and refusals.
tools/registry/README.md (calls table, "Not built yet"), RELEASE.md 7.9 and the write service's "as built" notes (7.11) updated: search is built over /api/1; ranking on the registry's package checks stays open until those checks exist.

Seed impact

None: TorbScript in std/project, compiler/src/project, compiler/src/package and tools/registry on existing std; no syntax change, no native, no driver flag, no std name the compiler looks up by string (an older torb ignores a keywords line in a lock's settings, as manifestOf skips unknown words).

Size and dependencies

One PR, two Conventional Commits: feat(project): keywords setting (std/project, manifest, lock, publish check, docs, skills) and feat(registry): search over /api/1 (schema, publish/yank/backfill, endpoint, tests, README, RELEASE.md). No blocking dependency. Overlap: the open PR #237 (documentation worker) and #223 (sessions and account page) also change tools/registry (database.trb migration, registry.trb, api.trb, main.trb, README, RELEASE.md rows); whichever lands second rebases, and the migrations are independent (each only adds tables).

Not in this issue

The search page: split into its own issue, depends on this one and on #12 (the registry's shell).
A torb search command: not asked by RELEASE.md; may be filed later together with torb login/owner (#22).
Ranking on package checks (documented declarations, lowers on every target, README, license): needs the data of #17 and #51.
Deploying or restarting the production registry (the backfill runs when its operator deploys the new image): outside the loop.

## Goal A visitor or a client can find packages in the registry by text: `GET /api/1/search` answers a ranked list over owner, name, description and keywords of what is published. The search page and a client command are not part of this issue (see "Not in this issue"). ## Source - [ROADMAP.md, the package manager](https://git.torb.dev/torbscript/language/src/commit/393ef2608e4a60b8ff76d8b6b408951b822d887d/docs/ROADMAP.md#L96) (open work: "the registry's sign-in, documentation worker, search and owners") - RELEASE.md section 7.9 "Search and ranking" (covers owner, name, description and `keywords`; **`keywords` is added**, not static, in the lock's `settings`, required by nothing; ranking after text relevance by dependents, then the registry's package checks) - RELEASE.md section 7.4 (the API carries search; "Description, keywords, the README and the documentation are on the API and the page", the signed index keeps only what resolution needs) and 7.11 ("Not built yet: ... search") - `tools/registry/README.md`, "Not built yet" ## Findings from refine - The registry stores neither description nor keywords: `releases` has only package, version, hash, publisher, yank state; the index record is signed and resolution-only, and stays exactly as it is. The description is in the archive's `project.lock.trb` `settings` block, which `checkedUpload` already reads on every publish. - `keywords` is not in the vocabulary yet (`std/project` `Project`, `Manifest`, the lock's `settings`); every other setting 7.9 needs exists. - RELEASE.md does **not** ask for `torb search` (7.13 lists `add`, `remove`, `update`, `install`, `publish`; 7.13 "Not built yet" names `yank`, `owner`, `login`; nothing names `search`). The acceptance bullet "`torb search` ... if RELEASE.md asks for it" therefore resolves to: not built here. - Neither the capability summary, the documentation worker nor the "checks" of 7.9 (documented declarations, lowers on every target, README, license) exist as registry data yet (#51, #17/#237), so ranking on checks cannot be built now. ## Design (decided by the loop, reasoning in the comment below) - **Vocabulary:** `keywords "http", "client"` in `project.trb`, variadic like `authors`: a variadic method of `Project` in `std/project`, `Manifest.keywords`, read by `manifestOf` and written by `settingsLines` into the lock's `settings`, listed in `unreadWords` (never evaluated by itself). `publishProblems` refuses at publish: more than 5 keywords, a keyword that is not 2 to 24 characters of lowercase `a-z`, `0-9`, `-` starting with a letter or digit, or a duplicate. Nothing else changes for a package without keywords. - **Data:** a derived table `search` (package, owner, name, version, description, keywords, capabilities, published) holding one row per package that has at least one non-yanked release, plus `search_dependencies (package, dependency)` for the runtime dependencies of that same release. The row describes the **latest non-yanked release** (highest stable version; the highest pre-release only when no stable one is published). It is rewritten in the same statement batch as the release row on publish and on yank/unyank, under the existing `WriteLock`; a package whose releases are all yanked has no row. The index is the truth and is not touched; the table is derived and can be rebuilt. - **Backfill:** at start, every package that has releases but no `search` row is filled from the archive of its latest non-yanked release (unpacked and read the way `checkedUpload` does); a package that fails is logged and skipped, never fatal. - **Matching and ranking:** the query is split on whitespace, lower-cased, at most 8 terms of at most 64 characters; a package must match every term (AND). Per term the best of: exact name 16, name prefix 8, name contains 4, keyword equal 4, owner equal 2, description contains 1 (substring, `instr` over lower-cased text); the sum is the relevance. Order: relevance, then dependents (distinct packages whose latest release lists it as a runtime dependency), then newest publish, then package name. A term with `/` is matched against `owner/name`. Downloads are not ranked on (7.9). - **API:** `GET /api/1/search?q=<text>&limit=<1..50, default 20>&offset=<default 0>`, public, no token. Answer `200`: `{"query": "...", "total": N, "limit": N, "offset": N, "packages": [{"package": "acme/http", "owner": "acme", "name": "http", "version": "1.2.4", "description": "...", "keywords": ["http"], "capabilities": ["network"], "dependents": 3, "published": 1790000000}]}` (integers are seconds since 1970 like the other calls). An empty `q` or one without a term, a bad `limit` or `offset`: `400` with the usual `{"error", "problems"}`. `Cache-Control: public, max-age=60`. - **Why a scan and not FTS5:** the registry reaches SQLite through the `sqlite3` shell, whose build options are the container's and the CI's business; an `instr` scan over the derived table needs none. Measured with SQLite 3.40 on 20 000 synthetic packages (12 words of description, 4 keywords each): 18 ms per query including the client; the registry will hold orders of magnitude fewer for a long time. Revisit only when a measurement shows more than 100 ms. ## Acceptance criteria - [ ] `keywords` exists in `std/project`, `Manifest`, the manifest reader and the lock's `settings` (round trip through `torb lock` and `torb lock --check`); `torb publish --dry-run` and the registry refuse the keyword cases above with one sentence each; a lock without `keywords` reads and writes exactly as before. `docs/tooling/project-trb.md` and the settings table of `docs/design/PROJECT.md` list it; `skills/` regenerated. - [ ] `tools/registry` migrates an existing database in place (new tables created, existing accounts, tokens and releases untouched) and backfills the `search` rows at start. - [ ] A publish writes the package's `search` row from the lock's `settings` of the archive; a newer version replaces it; a yank of the latest version moves the row to the next newest non-yanked one, a yank of the only version removes it, an unyank restores it; the signed index and the archives are byte-identical to what they were before this change for the same inputs. - [ ] `GET /api/1/search` matches owner, name, description and keywords, requires every term, orders by the rules above (tests with fixtures that pin each tier: exact name above prefix above contains above keyword above description; equal relevance ordered by dependents), honours `limit` and `offset`, reports `total`, returns an empty `packages` list for no match, and refuses the malformed requests with `400`. - [ ] A term that is SQL-hostile (`'`, `%`, `_`, `;`, control characters, 64+ characters) is matched literally or refused, never executed (test). - [ ] `tests/server.test.trb` (or a new test file) covers each case once as it works and once as it is refused; `torb test tools/registry/tests` green natively and in the VM; `compiler` tests for the `keywords` round trip and refusals. - [ ] `tools/registry/README.md` (calls table, "Not built yet"), RELEASE.md 7.9 and the write service's "as built" notes (7.11) updated: search is built over `/api/1`; ranking on the registry's package checks stays open until those checks exist. ## Seed impact None: TorbScript in `std/project`, `compiler/src/project`, `compiler/src/package` and `tools/registry` on existing `std`; no syntax change, no native, no driver flag, no std name the compiler looks up by string (an older `torb` ignores a `keywords` line in a lock's `settings`, as `manifestOf` skips unknown words). ## Size and dependencies One PR, two Conventional Commits: `feat(project): keywords setting` (std/project, manifest, lock, publish check, docs, skills) and `feat(registry): search over /api/1` (schema, publish/yank/backfill, endpoint, tests, README, RELEASE.md). No blocking dependency. Overlap: the open PR #237 (documentation worker) and #223 (sessions and account page) also change `tools/registry` (`database.trb` migration, `registry.trb`, `api.trb`, `main.trb`, README, RELEASE.md rows); whichever lands second rebases, and the migrations are independent (each only adds tables). ## Not in this issue - The search page: split into its own issue, depends on this one and on #12 (the registry's shell). - A `torb search` command: not asked by RELEASE.md; may be filed later together with `torb login`/`owner` (#22). - Ranking on package checks (documented declarations, lowers on every target, README, license): needs the data of #17 and #51. - Deploying or restarting the production registry (the backfill runs when its operator deploys the new image): outside the loop.

orion added this to the Milestone 8: formatter, linter, language server, package manager milestone

2026-10-05 21:07:10 +00:00

orion added the

loop

label

2026-10-05 21:07:10 +00:00

orion commented

2026-10-06 16:16:13 +00:00

Author

Collaborator

[loop:torbscript] intake: in scope (author orion, label loop). Registry search over the index via /api/1; the ranking/matching rules and whether torb search is wanted are a library design choice if RELEASE.md does not fix them, to settle in refine (possible owner decision). Deploy is outward-facing, not done by the loop. Release freeze for 0.1.0 (2026-10-06) holds new PRs, next: refine (queued)

[loop:torbscript] intake: in scope (author orion, label loop). Registry search over the index via `/api/1`; the ranking/matching rules and whether `torb search` is wanted are a library design choice if RELEASE.md does not fix them, to settle in refine (possible owner decision). Deploy is outward-facing, not done by the loop. Release freeze for 0.1.0 (2026-10-06) holds new PRs, next: refine (queued)

orion commented

2026-10-09 01:14:40 +00:00

Author

Collaborator

[loop:torbscript] claim refine by session:loop-torbscript-20261009 until 2026-10-09T03:14:00Z

orion referenced this issue

2026-10-09 01:17:24 +00:00

registry: search page #241

orion commented

2026-10-09 01:17:24 +00:00

Author

Collaborator

[loop:torbscript] decided by the loop: (1) Search is a derived search table scanned with instr, not FTS5 and not the signed index. The index stays resolution-only and signed (7.4); description and keywords come from the archive's lock settings at publish and are stored in SQLite, rewritten with the release row under the existing write lock, backfilled at start. A scan needs no SQLite build option (the registry uses the sqlite3 shell); measured 18 ms per query on 20 000 synthetic packages (SQLite 3.40, client included), revisit above 100 ms. (2) Ranking follows 7.9: text relevance (exact name 16, prefix 8, contains 4, keyword 4, owner 2, description 1, AND over terms), then dependents, then newest publish, then name; ranking on package checks waits for their data (#17, #51). (3) API: GET /api/1/search?q&limit&offset, public, JSON in the style of the other calls (seconds since 1970, {"error","problems"} refusals), limit 1..50 default 20, short Cache-Control. (4) keywords: variadic like authors, non-static, in the lock's settings, at most 5 of 2..24 characters [a-z0-9-], checked at publish only (crates.io-like limits against spam; no published package has any, so no contract breaks; an older torb ignores the line). (5) Search page split to #241 (static page of the site over the API, no server-rendered HTML in the write service); torb search is not asked by RELEASE.md and is not built. Out of scope: deploying or restarting the production registry. Overlap: PR #237 and #223 also change tools/registry; migrations are additive, the later one rebases, refine does not wait.

[loop:torbscript] decided by the loop: (1) **Search is a derived `search` table scanned with `instr`, not FTS5 and not the signed index.** The index stays resolution-only and signed (7.4); description and keywords come from the archive's lock `settings` at publish and are stored in SQLite, rewritten with the release row under the existing write lock, backfilled at start. A scan needs no SQLite build option (the registry uses the `sqlite3` shell); measured 18 ms per query on 20 000 synthetic packages (SQLite 3.40, client included), revisit above 100 ms. (2) **Ranking** follows 7.9: text relevance (exact name 16, prefix 8, contains 4, keyword 4, owner 2, description 1, AND over terms), then dependents, then newest publish, then name; ranking on package checks waits for their data (#17, #51). (3) **API:** `GET /api/1/search?q&limit&offset`, public, JSON in the style of the other calls (seconds since 1970, `{"error","problems"}` refusals), `limit` 1..50 default 20, short `Cache-Control`. (4) **`keywords`:** variadic like `authors`, non-static, in the lock's `settings`, at most 5 of 2..24 characters `[a-z0-9-]`, checked at publish only (crates.io-like limits against spam; no published package has any, so no contract breaks; an older `torb` ignores the line). (5) **Search page** split to #241 (static page of the site over the API, no server-rendered HTML in the write service); **`torb search`** is not asked by RELEASE.md and is not built. Out of scope: deploying or restarting the production registry. Overlap: PR #237 and #223 also change `tools/registry`; migrations are additive, the later one rebases, refine does not wait.

orion added the

ready

label

2026-10-09 01:17:24 +00:00

orion commented

2026-10-09 01:17:25 +00:00

Author

Collaborator

[loop:torbscript] refine: checked the requirements against ROADMAP, RELEASE.md 7.4/7.9/7.11/7.13, CONCEPT and tools/registry on main; the issue body now has goal, design, testable acceptance criteria, seed impact (none) and size (one PR, two Conventional Commits: feat(project): keywords setting, feat(registry): search over /api/1). The search page is split off, torb search is not asked by RELEASE.md (not built), deploy is out of scope; choices are recorded in the decided-by-the-loop comment. No owner decision needed. Overlap with #237 and #223 noted, not waited for. next: implement (ready). Worker usage: about 90k tokens (sonnet).

[loop:torbscript] refine: checked the requirements against ROADMAP, RELEASE.md 7.4/7.9/7.11/7.13, CONCEPT and `tools/registry` on `main`; the issue body now has goal, design, testable acceptance criteria, seed impact (none) and size (one PR, two Conventional Commits: `feat(project): keywords setting`, `feat(registry): search over /api/1`). The search page is split off, `torb search` is not asked by RELEASE.md (not built), deploy is out of scope; choices are recorded in the decided-by-the-loop comment. No owner decision needed. Overlap with #237 and #223 noted, not waited for. next: implement (ready). Worker usage: about 90k tokens (sonnet).

orion commented

2026-10-09 02:31:52 +00:00

Author

Collaborator

[loop:torbscript] claim implement by session:loop-torbscript-20261009 until 2026-10-09T08:31Z

orion referenced this issue from a commit

2026-10-09 02:52:12 +00:00

feat(registry): search over /api/1

orion referenced this issue from a commit

2026-10-09 02:52:59 +00:00

feat(registry): search over /api/1

orion referenced this issue from a pull request that will close it,

2026-10-09 02:53:13 +00:00

feat: registry search over /api/1 and the keywords setting #245

orion referenced this issue

2026-10-09 02:58:05 +00:00

repl: configurable limits and environment for --sandbox #49

orion referenced this issue

2026-10-09 02:58:06 +00:00

repl: configurable limits and environment for --sandbox #49

orion referenced this issue

2026-10-09 03:57:24 +00:00

registry: sessions, linked identities and the account page (slice 1 of #16) #223

orion commented

2026-10-09 04:14:46 +00:00

Author

Collaborator

PR: #245 (CI green on 5f71106, review requested from argus).

PR: https://git.torb.dev/torbscript/language/pulls/245 (CI green on 5f71106, review requested from argus).

argus closed this issue

2026-10-09 04:14:46 +00:00

argus referenced this issue from a commit

2026-10-09 04:14:48 +00:00

feat(registry): search over /api/1

orion commented

2026-10-09 04:15:09 +00:00

Author

Collaborator

[loop:torbscript] implement: PR #245 CI green on 5f71106 after two tier A reruns (each red run failed on one different timing-dependent test in compiler/tests/language-server.test.trb, tracked in #244; the PR touches no language-server code), approved and merged by argus. next: close (main CI on the landed commit)

[loop:torbscript] implement: PR https://git.torb.dev/torbscript/language/pulls/245 CI green on 5f71106 after two tier A reruns (each red run failed on one different timing-dependent test in `compiler/tests/language-server.test.trb`, tracked in #244; the PR touches no language-server code), approved and merged by argus. next: close (main CI on the landed commit)

orion commented

2026-10-09 04:15:26 +00:00

Author

Collaborator

[loop:torbscript] PR #245 merged by argus (911b10d). Issue closed by the merge. No after-merge steps in the PR; the registry deploy is outward-facing and not done by the loop.

[loop:torbscript] PR #245 merged by argus (911b10d). Issue closed by the merge. No after-merge steps in the PR; the registry deploy is outward-facing and not done by the loop.

orion commented

2026-10-09 04:38:54 +00:00

Author

Collaborator

[loop:torbscript] close: landed in 911b10d25d (PR #245); main CI is green on that commit (13 checks success, 11 platform/seed lanes skipped for this change). Issue closed.

[loop:torbscript] close: landed in https://git.torb.dev/torbscript/language/commit/911b10d25d085028f1a4a61d1c8293254734125d (PR #245); main CI is green on that commit (13 checks success, 11 platform/seed lanes skipped for this change). Issue closed.

orion referenced this issue

2026-10-09 05:46:21 +00:00

registry: search page #241

orion referenced this issue

2026-10-10 10:45:35 +00:00

registry: search page #241