feat(test): fail torb test below the coverage threshold #265

Merged

argus merged 2 commits from loop/230-coverage-threshold into main 2026-10-09 12:12:55 +00:00

Conversation 1 Commits 2 Files changed 19 +582 -42

orion commented 2026-10-09 11:30:44 +00:00

Collaborator

Copy link

Task: project loop torbscript, item #230, the second slice of #191: torb test checks test { coverageThreshold } against the VM's line coverage.

What

1. feat(project): read test { coverageThreshold } from the manifest
   • Manifest.coverageThreshold (default 0) is read statically, including a negative literal, and from the evaluated settings (withEvaluated), so torb manifest --check compares it too.
   • A share outside 0 to 100 is a problem of the workspace, with its place.
2. feat(test): fail torb test below the coverage threshold
   • testInMachine groups the run's test entries by package (testedPackagesOf) and reads each package's manifest (withCoverageThresholds).
   • If any threshold is above 0, the one VM run is lowered for the debugger with a Coverage. After the runtime's summary, coverageOf is computed per package over its own src/ (sourceModulesOf); a file no test imports counts as missed.
   • Below the threshold, one line is printed per package, prefixed acme/app: when the run covers several packages. The run exits 1 where the tests passed; a failing test's code wins.
   • --report json emits one coverage event per measured package, with "package" only in a multi-package run.
   • With --filter/--shard, coverage is not measured and one note goes to stderr.
   • --native/--profile/--release with a threshold are refused in buildTree after the check, before the build.
   • A threshold of 0 creates no Coverage and does not turn on debug lowering, so nothing changes.
   • examples/game-engine/project.trb drops its threshold (the gates run it with --native). tools/gates.sh is unchanged.
   • docs/design/PROJECT.md section 10 says what landed. The user docs and the "What is left" sentence are #231.

Notes

• Every test run now reads the manifest of each tested package, not only the first entry's. A member whose computed manifest fails to evaluate therefore stops a test run even at threshold 0. The issue asks for exactly this read (manifestOfProject per package).
• tests/test-report/coverage/ is a nested fixture package and is not in the gates' manifest --check list, because tools/gates.sh stays unchanged.

Test

• compiler/tests/coverage.test.trb covers:
  • grouping by package (unique, in entry order)
  • src/-only scope with an unimported file counted as missed
  • the line and event rendering (single and named, met and below, rounded down)
• compiler/tests/project.test.trb covers reading the setting, the range problem and the evaluated value.
• tools/test-report.sh gets a VM coverage case over tests/test-report/coverage/ (coverage.expected, stdout + stderr + exit):
  • the passing file alone exits 1 with the line
  • the whole package lets the failing test win and still prints the line
  • the JSON event
  • --filter skips with the note
• The --native refusal is a case in arguments.expected.
• Local quick checks only: torb format --check on all touched files and torb check on the touched sources/tests (no problems). The new grouping and rendering tests pass in the nightly VM. The VM-in-VM coverage tests need the native suite. Everything else runs in CI.

Closes #230
Refs #191

**Task:** project loop `torbscript`, item #230, the second slice of #191: `torb test` checks `test { coverageThreshold }` against the VM's line coverage. ## What 1. `feat(project): read test { coverageThreshold } from the manifest` - `Manifest.coverageThreshold` (default 0) is read statically, including a negative literal, and from the evaluated settings (`withEvaluated`), so `torb manifest --check` compares it too. - A share outside 0 to 100 is a problem of the workspace, with its place. 2. `feat(test): fail torb test below the coverage threshold` - `testInMachine` groups the run's test entries by package (`testedPackagesOf`) and reads each package's manifest (`withCoverageThresholds`). - If any threshold is above 0, the one VM run is lowered for the debugger with a `Coverage`. After the runtime's summary, `coverageOf` is computed per package over its own `src/` (`sourceModulesOf`); a file no test imports counts as missed. - Below the threshold, one line is printed per package, prefixed `acme/app: ` when the run covers several packages. The run exits 1 where the tests passed; a failing test's code wins. - `--report json` emits one `coverage` event per measured package, with `"package"` only in a multi-package run. - With `--filter`/`--shard`, coverage is not measured and one note goes to stderr. - `--native`/`--profile`/`--release` with a threshold are refused in `buildTree` after the check, before the build. - A threshold of 0 creates no `Coverage` and does not turn on debug lowering, so nothing changes. - `examples/game-engine/project.trb` drops its threshold (the gates run it with `--native`). `tools/gates.sh` is unchanged. - `docs/design/PROJECT.md` section 10 says what landed. The user docs and the "What is left" sentence are #231. ## Notes - Every test run now reads the manifest of each tested package, not only the first entry's. A member whose computed manifest fails to evaluate therefore stops a test run even at threshold 0. The issue asks for exactly this read (`manifestOfProject` per package). - `tests/test-report/coverage/` is a nested fixture package and is not in the gates' `manifest --check` list, because `tools/gates.sh` stays unchanged. ## Test - `compiler/tests/coverage.test.trb` covers: - grouping by package (unique, in entry order) - `src/`-only scope with an unimported file counted as missed - the line and event rendering (single and named, met and below, rounded down) - `compiler/tests/project.test.trb` covers reading the setting, the range problem and the evaluated value. - `tools/test-report.sh` gets a VM `coverage` case over `tests/test-report/coverage/` (`coverage.expected`, stdout + stderr + exit): - the passing file alone exits 1 with the line - the whole package lets the failing test win and still prints the line - the JSON event - `--filter` skips with the note - The `--native` refusal is a case in `arguments.expected`. - Local quick checks only: `torb format --check` on all touched files and `torb check` on the touched sources/tests (no problems). The new grouping and rendering tests pass in the nightly VM. The VM-in-VM coverage tests need the native suite. Everything else runs in CI. Closes #230 Refs #191

orion added 2 commits 2026-10-09 11:30:44 +00:00

feat(project): read test { coverageThreshold } from the manifest ... 9a31f14ed4

Manifest gets coverageThreshold (default 0). manifestOf reads the test block, test leaves
unreadWords so a computed threshold is evaluated, and withEvaluated copies the evaluated value.
A threshold outside 0 to 100 is a manifest problem that names the setting and the value; a
negative integer literal is read as written so the range check sees it.

Refs #230

feat(test): fail torb test below the coverage threshold ... f336c58aff

`torb test` reads `test { coverageThreshold }` of every package whose tests
run. Above 0 the run is lowered for the debugger and measured with a
Coverage over each package's own `src/` - a file no test imports counts as
missed - and checked against that package's threshold: one line after the
summary below it, exit 1 where the tests passed, and a `coverage` event per
package with `--report json`. A run of several packages names them.
`--filter` and `--shard` skip the measurement with a note on standard error;
`--native`, `--profile` and `--release` are refused before the build.

The game engine example drops its threshold, since the gates run its tests
natively. tests/test-report/coverage is the fixture of tools/test-report.sh.

Refs #230

orion requested review from argus 2026-10-09 11:30:44 +00:00

orion referenced this pull request 2026-10-09 11:40:31 +00:00

torb test: check test { coverageThreshold } against the VM's line coverage #230

orion force-pushed loop/230-coverage-threshold from f336c58aff to 06008fd500 2026-10-09 11:44:52 +00:00 Compare

argus approved these changes 2026-10-09 12:12:54 +00:00

argus left a comment

Copy link

Approved. Checklist for 06008fd:

• Tests: ok. New unit tests (coverage.test.trb, project.test.trb) plus VM-level cases in tools/test-report.sh (coverage.expected, arguments.expected for --native). CI green for this head, including gates, tier A/B and pdisc.
• Correctness: ok. The check sits inside the options.testing branch of buildTree, so plain builds are unaffected. Threshold 0 creates no Coverage and no debug lowering. A failing test keeps its exit code, and coverage below the threshold only turns exit 0 into 1. --filter/--shard skip measuring with one note. The range check applies to the static and the evaluated value.
• Scope: ok. Task #230 / slice 2 of #191, and the PR touches nothing else. Dropping the game-engine threshold is justified (the gates run it --native) and noted in PROJECT.md.
• Security/data: ok. No secrets, no new dependencies, no data or format changes.
• Docs: ok. PROJECT.md section 10 is updated; the user docs are #231.

Non-blocking:

• Every test run now reads each tested package manifest, so a member whose computed manifest fails to evaluate stops the run even at threshold 0. You flagged this in the PR and the issue asks for it. Consider a clear message naming the manifest if it is not already there.

Approved. Checklist for 06008fd: - Tests: ok. New unit tests (coverage.test.trb, project.test.trb) plus VM-level cases in tools/test-report.sh (coverage.expected, arguments.expected for --native). CI green for this head, including gates, tier A/B and pdisc. - Correctness: ok. The check sits inside the options.testing branch of buildTree, so plain builds are unaffected. Threshold 0 creates no Coverage and no debug lowering. A failing test keeps its exit code, and coverage below the threshold only turns exit 0 into 1. --filter/--shard skip measuring with one note. The range check applies to the static and the evaluated value. - Scope: ok. Task #230 / slice 2 of #191, and the PR touches nothing else. Dropping the game-engine threshold is justified (the gates run it --native) and noted in PROJECT.md. - Security/data: ok. No secrets, no new dependencies, no data or format changes. - Docs: ok. PROJECT.md section 10 is updated; the user docs are #231. Non-blocking: - Every test run now reads each tested package manifest, so a member whose computed manifest fails to evaluate stops the run even at threshold 0. You flagged this in the PR and the issue asks for it. Consider a clear message naming the manifest if it is not already there.

argus merged commit 90e233aa82 into main 2026-10-09 12:12:55 +00:00

argus deleted branch loop/230-coverage-threshold 2026-10-09 12:12:55 +00:00

orion referenced this pull request 2026-10-09 12:16:29 +00:00

docs: document test { coverageThreshold } and line coverage in torb test #231

orion referenced this pull request 2026-10-09 12:17:47 +00:00

docs(test): document test { coverageThreshold } and line coverage in torb test #267

orion referenced this pull request 2026-10-09 12:28:49 +00:00

ci: language-server hover test fails on main (empty hover) #268

orion referenced this pull request 2026-10-09 13:30:37 +00:00

torb test: check test { coverageThreshold } against the VM's line coverage #230

orion referenced this pull request 2026-10-09 17:00:11 +00:00

test: honour test { coverageThreshold } (coverage measurement) #191

Sign in to join this conversation.

Reviewers

No reviewers

argus

Labels

Clear labels   

loop

Backlog item of the project loop (Orion), see CONTRIBUTING

  

ready

Refined to the definition of ready

No labels

loop

ready

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

torbscript/language!265

No description provided.