torb test: check test { coverageThreshold } against the VM's line coverage #230
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Second slice of #191 (option A); the measurement is in #222 (
Coverage,coverageOf,CoverageReport.meets, decision record indocs/design/PROJECT.mdsection 10, "test { coverageThreshold }, line coverage in the VM"). Docs follow in #231.Acceptance criteria
One PR, two Conventional Commits. No seed impact (see the loop's comment below).
Commit 1,
feat(project): read test { coverageThreshold } from the manifestManifest(compiler/src/project/manifest.trb) hascoverageThreshold: Int(default0);manifestOfreads thetest { coverageThreshold = N }block;testleavesunreadWords, so a computed threshold makesneedsEvaluationtrue;withEvaluated(compiler/src/vm/manifest.trb) copies the evaluated value.0..100is a manifest problem naming the setting and the value (section 10's "Error" column; nothing enforces it today).compiler/tests/project.test.trb: literal, computed (evaluated), default0, and the out-of-range problem.Commit 2,
feat(test): fail torb test below the coverage thresholdtestInMachine(compiler/src/vm/run.trb): for every package whose tests run, read its manifest (manifestOfProject); when any has a threshold above 0, lower for the debugger, run with aCoverage, and computecoverageOfover that package'ssrc/modules. Each package is checked against its own threshold (a multi-path run such as the gates'torb test <many packages>is one VM run with one report per package). Asrc/file that no test imports counts as missed (test for it).coverage 71% is below the threshold 80%(with several packages in the run the line is prefixed by the package name:acme/app: coverage 71% is below the threshold 80%); exit code 1 when the tests passed; a failing test's exit code wins and the line is still printed. Share rounded down, compared withmeets.--report json: onecoverageevent per measured package after the summary,{"event":"coverage","percent":71,"threshold":80,"lines":120,"reached":86}(plus"package"only in a multi-package run; emitted whether or not the threshold is met).--filterand--shardwith a threshold above 0: coverage is not measured (a partial run measures nothing) and one line on standard error says so; nothing fails.runTests(compiler/src/cli/test.trb):--native(or--profile/--release, which build natively) with any package at a threshold above 0 is an error before anything is built, exit 1:error: `test { coverageThreshold = 80 }` of acme/app is measured in the VM: run `torb test` without `--native`.0(default): no lowering for the debugger, no new output, same exit code as today (existingtools/test-report.shexpectations unchanged).examples/game-engine/project.trbdrops itstest { coverageThreshold = 80 }block (see the loop's decision);tools/gates.shis unchanged.tests/test-report/(or beside the suite), run bytools/test-report.shin the VM for the message, exit code, JSON event, and a failing test beating the coverage code; the--nativeerror as a case ofarguments.expectedor a native case; unit tests incompiler/testsfor the multi-package grouping.docs/design/PROJECT.mdsection 10 (the slice it calls next, the game-engine remark) says what landed; the user docs and the "What is left" sentence are #231.Refs #191
test { coverageThreshold }(coverage measurement) #191test { coverageThreshold }(coverage measurement) #191[loop:torbscript] intake: in scope (author orion, label loop), second slice of #191 on top of the measurement in #222 (landed), not a duplicate. Wiring in torb test. next: waiting (refine queued, ready buffer full) until 2026-10-10T04:00:00Z
[loop:torbscript] claim refine by session:loop-torbscript-20261009 until 2026-10-09T13:04Z
[loop:torbscript] decided by the loop: the open choices of #191's option A, with the numbers I could read without running anything.
Message, exit code, JSON event. As the design record (
docs/design/PROJECT.mdsection 10) and the issue say: one linecoverage 71% is below the threshold 80%after the summary, exit code 1 (the same code a failing test andrefusedCodeuse; a failing test wins, the line is still printed), onecoverageJSON event after the summary. With several packages in one run (the gates runtorb test <all std/example packages>as one VM run) each package is checked against its own manifest, as the record says for workspace members; the line gets a<package>:prefix and the event a"package"field only in that case. The event is emitted whenever coverage was measured, not only on failure.Reading the setting. Nothing reads it:
Manifesthas no field,testis inunreadWords(compiler/src/project/manifest.trb:553), and the range0..100of section 10's error column is enforced nowhere (grep overcompiler/src). The settings string the evaluation writes already carriestest { coverageThreshold }(std/project/src/lib.trb:257), so the existingmanifestOfProject/withEvaluatedpath (as fortasksandprofile) is the mechanism. That makes the first commit.--native. An error before anything is built, exit 1, naming the setting, its value and the package, and saying coverage is measured in the VM. Same for--profile/--release(they build natively). Never silently ignored, per the owner decision.--filter/--shard. Not in the #191 comment, but needed: a partial run would read far below the threshold and fail every sharded CI job. Decision: not measured, one line on stderr, nothing fails. An error would break the everydaytorb test --filter xfor any project with a threshold.The game-engine manifest and the native gate: drop the threshold, leave the gate alone. Reasons: (1) The example's tests (
world.test.trb, 3 + 3 tests) call onlymovementandcleanupofsystems.trb; counting by hand, about 7 of about 27 statement lines ofsystems.trband none of therunfunction ofmain.trbare reached, so the example is far below 80% (estimate by reading, not a measurement: coverage is not wired yet, and I ran no build). With the threshold kept, the VM gatetest (std/example packages, VM)would go red the moment this lands. (2) Keeping it and skipping game-engine in the native gate (brokenlist) costs the native test of the one example that exercises the scene DSL and closures, which RUST-EXIT 2.4 wants built natively. (3) Lifting the example to 80% means new tests, which is a different item. The value 80 was never read by anything, so dropping it changes no behaviour; the feature gets its own fixture package undertests/test-report/, run by the existingtools/test-report.shgate (VM and--nativeerror), so the check is exercised in CI without touching the example.std/prelude/project.trbalso setscoverageThreshold = 90; it has notests/directory, no gate runs it, so it stays untouched. No other manifest in the repository sets the setting (grep over allproject.trb). This is an internal repository choice, so no owner question; Torben may of course re-add a threshold to an example later.Unimported source files.
coverageOfcounts the modules it is given; the implementer must hand it everysrc/file of the package, not only the ones a test imports, and test it (a file nobody imports is all missed), otherwise a threshold could be gamed by leaving code unimported.check ... onlyReached: trueparses the packages the program reaches, which should cover the whole package; the test settles it.Seed impact: none. No syntax change, no new native, no new driver flag (the command line is unchanged), no std name the compiler looks up by string (
Test.coverageThresholdand thetestblock already exist instd/project);Manifestgains a field incompiler/src/written in the language the seed already compiles. No seed refresh.Size. One PR, two commits: (1)
feat(project): read test { coverageThreshold } from the manifest(Manifest field, evaluation, range problem, tests inproject.test.trb); (2)feat(test): fail torb test below the coverage threshold(run.trb, cli/test.trb, game-engine manifest, fixture and expectations intests/test-report/, PROJECT.md closing paragraph). Docs stay in #231. Implementer: opus/high (compiler/srcVM and project code). The gate that matters is forge CI (tools/gates.sh a); the fixture's expected files come fromtools/test-report.sh --updateread in CI, not from a local run.I replaced the acceptance list in the issue description with the tightened one (original wording of the goal and the references kept).
[loop:torbscript] refine: ready, decisions recorded above (one PR, two commits, no seed impact, no owner question), claim released, next: ready (implement)
[loop:torbscript] claim implement by session:loop-torbscript-20261009 until 2026-10-09T17:08Z
test { coverageThreshold }(coverage measurement) #191const r: RingList<Int> = [1, 2]) #212[loop:torbscript] implement: PR #265 opened (two commits,
feat(project)thenfeat(test), review requested from argus); the worker ended before reporting. CI onf336c58a: tier A (linux-x64) red, gates and tier B still running. next: fix (tier A)[loop:torbscript] claim fix by session:loop-torbscript-20261009 until 2026-10-09T17:40Z
[loop:torbscript] fix: tier A failed in
compiler/tests/coverage.test.trb("a package is measured over its src/ alone ...",lib.reached == [2]). The fixtureapp/src/main.trbheld only auseand a named top-levelconst, soisEntryModule(compiler/src/ir/lower/function.trb) did not treat it as an entry file, no entry function was lowered, and the run reached nothing. The fixture now reads the constant in anif(as the first group's program does), which makes it an entry file; folded intofeat(test), still two Conventional Commits on current main,tools/check-commits.shclean. Not a flake, no rerun. CI green on06008fd5(tier A, tier B, gates, every host emits the same C, playground, VS Code, pdisc). next: review (waiting on the merge gate argus). Tokens: ~127k.[loop:torbscript] close: PR #265 approved and merged by argus (
90e233aa); main CI is green on the later head2990ac8f(tier A flake of an unrelated LSP test fixed in #268). Claim released.test { coverageThreshold }(coverage measurement) #191