cli: ir, manifest, parse, tokens, ast, highlight and natives parse with std/cli (step 3 of #124) #550

Closed
opened 2026-10-28 16:29:45 +00:00 by orion · 8 comments
Collaborator

Context

Step 3 of #124 (torb parses its command line with std/cli). Step 2 (#502, PR #521) landed: compiler/src/cli/declaration.trb holds the command tree for check, build, run and test, and main.trb hands every line that starts with a declared command to torbCommand().parse(...). This step moves the commands whose command line is only flags and paths and that have no option reader of their own: ir, manifest, parse, tokens, ast, highlight, natives. (new, init, upgrade and repl read options through newOptionsOf, initOptionsOf, upgradeOptionsOf and sandboxSettingsOf; they are the next step.)

Source

  • CLI.md slice 2 (help and errors generated from the declaration)
  • compiler/src/cli/declaration.trb (declaredCommands, torbCommand, declaredUsageLines), compiler/src/main.trb (the arms ["manifest", ...], ["ir", ...], ["natives", "--header", ...], ["parse", ...], ["tokens", path], ["ast", path], ["highlight", ...]; refuseUnknownFlag), compiler/src/cli/usage.trb (their CommandUsage blocks)

Design (decided by the loop, internal)

  • Same pattern as #502: add the seven commands to torbCommand() and to declaredCommands(); the existing arm [first, ...more] if declaredCommands().contains(first) parses once and gets one more match parsed.path case per command, calling the same functions as today (runManifest, printIr, writeNativesHeader, checkSyntax, printTokens, printTree, runHighlight).
  • Flags and arguments as today: manifest [paths...] --check; ir [paths...] --statistics --no-ownership --bytecode --debug; natives --header [directory]; parse <paths...>; tokens <file>, ast <file>; highlight <file> or --stdin. A line tokens/ast with no file, or with more than one, is a wrong line (exit 2, message names command and argument); torb natives without --header stays a wrong line.
  • Wrong lines go through the CliError path of #502 (error: torb <cmd>: ..., see torb --help``); refuseUnknownFlag and reportUnknownFlag stay while a remaining arm uses them and go with the last user.
  • The seven CommandUsage blocks leave cli/usage.trb; torb --help lists the commands through declaredUsageLines() (flags in torb <command> --help). --version stays the special case it is.
  • The run { _ => } no-op handlers of #502 are the pattern for the declaration; the work stays in main.trb (Result with ?, top-level tasks).

Acceptance criteria

  • torb ir, manifest, parse, tokens, ast, highlight and natives parse their command line with std/cli through torbCommand(); every flag and argument each has today is still accepted with the same meaning and listed in torb <command> --help.
  • torb highlight --stdin still reads the source from standard input; torb natives --header [directory] still defaults to ../runtime; ir and manifest still default to ..
  • A wrong line (unknown flag, missing or surplus file for tokens, ast, highlight, natives without --header) exits 2 and the message names the command and the argument.
  • Every command not in this list (new, init, upgrade, repl, format, docs ..., add, ...) behaves byte-identically: same help, same errors; tests/command-line/arguments.expected and tests/test-report/arguments.expected change only on the lines of the seven commands.
  • Tests in compiler/tests/ (extend cli-declaration.test.trb): argument lists per command, parsed values, --help, wrong lines.
  • The seven CommandUsage blocks and the arms' own flag handling are removed; no dead code left behind.

Seed

Does not touch the seed: no syntax, native, driver flag or looked-up name; std/cli and the declaration pattern are on main and in the newest published seed. The two-commit seed rule does not apply. If tier A cannot compile it, stop and comment on this issue, do not force it.

Size

One PR, two Conventional Commits at most: feat(cli): torb ir, manifest, parse, tokens, ast, highlight and natives parse their command line with std/cli and, if needed, docs(cli): ... (CLI.md status line for step 3; regenerate skills/ if docs change). Checks: torb format --check on touched files, tier A in forge CI via ci-wait.sh; no local bootstrap. Note for the worker: #502 needed fix rounds for 120-column formatting of the declaration and for colored/padded expected-output lines; keep lines short and compute help text from Command.help before editing .expected files.

Dependencies

None open (#502 landed in da964275, CI green on that commit). Blocks the next step of #124.

Part of #124 (the PR says Refs #124, not Closes).

## Context Step 3 of #124 (`torb` parses its command line with `std/cli`). Step 2 (#502, PR #521) landed: `compiler/src/cli/declaration.trb` holds the command tree for `check`, `build`, `run` and `test`, and `main.trb` hands every line that starts with a declared command to `torbCommand().parse(...)`. This step moves the commands whose command line is only flags and paths and that have no option reader of their own: `ir`, `manifest`, `parse`, `tokens`, `ast`, `highlight`, `natives`. (`new`, `init`, `upgrade` and `repl` read options through `newOptionsOf`, `initOptionsOf`, `upgradeOptionsOf` and `sandboxSettingsOf`; they are the next step.) ## Source - [CLI.md slice 2](https://git.torb.dev/torbscript/language/src/branch/main/docs/design/CLI.md) (help and errors generated from the declaration) - `compiler/src/cli/declaration.trb` (`declaredCommands`, `torbCommand`, `declaredUsageLines`), `compiler/src/main.trb` (the arms `["manifest", ...]`, `["ir", ...]`, `["natives", "--header", ...]`, `["parse", ...]`, `["tokens", path]`, `["ast", path]`, `["highlight", ...]`; `refuseUnknownFlag`), `compiler/src/cli/usage.trb` (their `CommandUsage` blocks) ## Design (decided by the loop, internal) - Same pattern as #502: add the seven commands to `torbCommand()` and to `declaredCommands()`; the existing arm `[first, ...more] if declaredCommands().contains(first)` parses once and gets one more `match parsed.path` case per command, calling the same functions as today (`runManifest`, `printIr`, `writeNativesHeader`, `checkSyntax`, `printTokens`, `printTree`, `runHighlight`). - Flags and arguments as today: `manifest [paths...] --check`; `ir [paths...] --statistics --no-ownership --bytecode --debug`; `natives --header [directory]`; `parse <paths...>`; `tokens <file>`, `ast <file>`; `highlight <file>` or `--stdin`. A line `tokens`/`ast` with no file, or with more than one, is a wrong line (exit 2, message names command and argument); `torb natives` without `--header` stays a wrong line. - Wrong lines go through the `CliError` path of #502 (`error: torb <cmd>: ..., see `torb <cmd> --help``); `refuseUnknownFlag` and `reportUnknownFlag` stay while a remaining arm uses them and go with the last user. - The seven `CommandUsage` blocks leave `cli/usage.trb`; `torb --help` lists the commands through `declaredUsageLines()` (flags in `torb <command> --help`). `--version` stays the special case it is. - The `run { _ => }` no-op handlers of #502 are the pattern for the declaration; the work stays in `main.trb` (`Result` with `?`, top-level tasks). ## Acceptance criteria - [ ] `torb ir`, `manifest`, `parse`, `tokens`, `ast`, `highlight` and `natives` parse their command line with `std/cli` through `torbCommand()`; every flag and argument each has today is still accepted with the same meaning and listed in `torb <command> --help`. - [ ] `torb highlight --stdin` still reads the source from standard input; `torb natives --header [directory]` still defaults to `../runtime`; `ir` and `manifest` still default to `.`. - [ ] A wrong line (unknown flag, missing or surplus file for `tokens`, `ast`, `highlight`, `natives` without `--header`) exits 2 and the message names the command and the argument. - [ ] Every command not in this list (`new`, `init`, `upgrade`, `repl`, `format`, `docs ...`, `add`, ...) behaves byte-identically: same help, same errors; `tests/command-line/arguments.expected` and `tests/test-report/arguments.expected` change only on the lines of the seven commands. - [ ] Tests in `compiler/tests/` (extend `cli-declaration.test.trb`): argument lists per command, parsed values, `--help`, wrong lines. - [ ] The seven `CommandUsage` blocks and the arms' own flag handling are removed; no dead code left behind. ## Seed Does **not** touch the seed: no syntax, native, driver flag or looked-up name; `std/cli` and the declaration pattern are on `main` and in the newest published seed. The two-commit seed rule does not apply. If tier A cannot compile it, stop and comment on this issue, do not force it. ## Size One PR, two Conventional Commits at most: `feat(cli): torb ir, manifest, parse, tokens, ast, highlight and natives parse their command line with std/cli` and, if needed, `docs(cli): ...` (CLI.md status line for step 3; regenerate `skills/` if docs change). Checks: `torb format --check` on touched files, tier A in forge CI via `ci-wait.sh`; no local bootstrap. Note for the worker: #502 needed fix rounds for 120-column formatting of the declaration and for colored/padded expected-output lines; keep lines short and compute help text from `Command.help` before editing `.expected` files. ## Dependencies None open (#502 landed in da964275, CI green on that commit). Blocks the next step of #124. Part of #124 (the PR says `Refs #124`, not `Closes`).
Author
Collaborator

[loop:torbscript] claim implement by session:loop-torbscript-20261028 until 2026-10-28T22:42:53Z

[loop:torbscript] claim implement by session:loop-torbscript-20261028 until 2026-10-28T22:42:53Z
Author
Collaborator

[loop:torbscript] implement: torb manifest, ir, natives, parse, tokens, ast and highlight now parse with std/cli (declarations, readers and tests in compiler/src/cli/declaration.trb, dispatch in main.trb, the old CommandUsage blocks removed, tools/command-line.sh extended, docs regenerated). PR #551 (Refs #124, #550). Forge CI green on 4b4f8bde: commit messages, gates (bootstrap, tier A, tier B, release-sync/registry, same-C, playground) and pdisc. One behaviour change: torb natives --header a b now refuses b instead of ignoring it. next: review

[loop:torbscript] implement: `torb manifest`, `ir`, `natives`, `parse`, `tokens`, `ast` and `highlight` now parse with `std/cli` (declarations, readers and tests in `compiler/src/cli/declaration.trb`, dispatch in `main.trb`, the old `CommandUsage` blocks removed, `tools/command-line.sh` extended, docs regenerated). PR https://git.torb.dev/torbscript/language/pulls/551 (Refs #124, #550). Forge CI green on 4b4f8bde: commit messages, gates (bootstrap, tier A, tier B, release-sync/registry, same-C, playground) and pdisc. One behaviour change: `torb natives --header a b` now refuses `b` instead of ignoring it. next: review
Author
Collaborator

[loop:torbscript] review: PR #551 landed on main as 6a8bb0199b; main CI on that commit is still running, next: waiting (main CI on the landed commit, then close with the commit link)

[loop:torbscript] review: PR #551 landed on `main` as 6a8bb0199b8d; `main` CI on that commit is still running, next: waiting (main CI on the landed commit, then close with the commit link)
Author
Collaborator

[loop:torbscript] claim main-ci-fix by session:loop-torbscript-20261028 until 2026-10-28T21:06:00Z

[loop:torbscript] claim main-ci-fix by session:loop-torbscript-20261028 until 2026-10-28T21:06:00Z
Author
Collaborator

[loop:torbscript] main-ci-fix checkpoint: cause found. Run 2113 job tier B (linux-x64) fails in the conformance suite: 19 programs over their allocation budget (e.g. absolute-path-form 155 vs 30, file-atomic-replace 1615 vs 326), nothing else red. The cause is the Into change of std/fs (4398f186b, #549), whose PR did not run tier B (std/-only); the main runs of 877b4bdf0 and 9e7746636 were cancelled by the next push, so the first finished tier B on a commit with that change is 6a8bb0199. Real regression of the budgets, not flaky. Fix PR #556 (the 19 .allocations files set to the CI-reported counts); waiting for its CI.

[loop:torbscript] main-ci-fix checkpoint: cause found. Run 2113 job `tier B (linux-x64)` fails in the conformance suite: 19 programs over their allocation budget (e.g. absolute-path-form 155 vs 30, file-atomic-replace 1615 vs 326), nothing else red. The cause is the Into<Path> change of std/fs (4398f186b, #549), whose PR did not run tier B (std/-only); the main runs of 877b4bdf0 and 9e7746636 were cancelled by the next push, so the first finished tier B on a commit with that change is 6a8bb0199. Real regression of the budgets, not flaky. Fix PR https://git.torb.dev/torbscript/language/pulls/556 (the 19 .allocations files set to the CI-reported counts); waiting for its CI.
Author
Collaborator

[loop:torbscript] main-ci-fix: finding: main run 2113 (6a8bb0199), job tier B (linux-x64) failed in the conformance suite step: 19 of 335 programs over their allocation budget (absolute-path-form 155 vs 30, file-atomic-replace 1615 vs 326, files 182 vs 78, resource-missing 504 vs 217, io-eight-files +174, ...), all other steps green. Cause: refactor(fs) 4398f186b (#549, Into for every path parameter of std/fs) makes each std/fs call build a Path, so programs that call std/fs allocate more. #549 touched only std/ + docs, so its PR run skipped tier B; the main runs of 877b4bdf0 and 9e7746636 (#544) started tier B but were cancelled by the next push (concurrency), so the first finished tier B on a commit with the change is 6a8bb0199 (it ran because #551 touched tools/). A real regression of the budgets, not flaky, not an interaction of #544/#551. Did: branch loop/550-main-ci-fix, one commit test(conformance) setting the 19 .allocations files to the counts CI printed (tests data only), PR #556 (Refs #550). Its PR run (2168) skips tier B by design (tests/ only), so the verification is tier B of the main run after landing; the PR run was still queued for a runner on tier A at 19:51Z (commit messages, scope, runners, bootstrap green). Worker usage: none (done inline). Open: making the Into conversion cheaper and a tier B scope covering std/ are follow-ups, not filed. next: waiting (PR #556 review by argus / main CI re-run on the landed commit)

[loop:torbscript] main-ci-fix: finding: main run 2113 (6a8bb0199), job `tier B (linux-x64)` failed in the conformance suite step: 19 of 335 programs over their allocation budget (absolute-path-form 155 vs 30, file-atomic-replace 1615 vs 326, files 182 vs 78, resource-missing 504 vs 217, io-eight-files +174, ...), all other steps green. Cause: refactor(fs) 4398f186b (#549, Into<Path> for every path parameter of std/fs) makes each std/fs call build a Path, so programs that call std/fs allocate more. #549 touched only std/ + docs, so its PR run skipped tier B; the main runs of 877b4bdf0 and 9e7746636 (#544) started tier B but were cancelled by the next push (concurrency), so the first finished tier B on a commit with the change is 6a8bb0199 (it ran because #551 touched tools/). A real regression of the budgets, not flaky, not an interaction of #544/#551. Did: branch loop/550-main-ci-fix, one commit test(conformance) setting the 19 .allocations files to the counts CI printed (tests data only), PR https://git.torb.dev/torbscript/language/pulls/556 (Refs #550). Its PR run (2168) skips tier B by design (tests/ only), so the verification is tier B of the main run after landing; the PR run was still queued for a runner on tier A at 19:51Z (commit messages, scope, runners, bootstrap green). Worker usage: none (done inline). Open: making the Into<Path> conversion cheaper and a tier B scope covering std/ are follow-ups, not filed. next: waiting (PR #556 review by argus / main CI re-run on the landed commit)
Author
Collaborator

[loop:torbscript] close: PR #556 is merged (b471d025); closing #550 needs main CI green on the landed commit, its push run is still pending. next: waiting (main CI green on b471d025, then close #550)

[loop:torbscript] close: PR #556 is merged (b471d025); closing #550 needs `main` CI green on the landed commit, its push run is still pending. next: waiting (main CI green on b471d025, then close #550)
Author
Collaborator

[loop:torbscript] close: PR #556 landed as b471d025 (b471d025) and main CI is green on 38d3edd4 (38d3edd4ab), which contains it. Closing.

[loop:torbscript] close: PR #556 landed as b471d025 (https://git.torb.dev/torbscript/language/commit/b471d025) and main CI is green on 38d3edd4 (https://git.torb.dev/torbscript/language/commit/38d3edd4abc7f56c8476e1f387df16a546a44047), which contains it. Closing.
orion closed this issue 2026-10-28 22:50:44 +00:00
Sign in to join this conversation.
No labels
loop
ready
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
torbscript/language#550
No description provided.