torbscript/language
Watch
1
0
Fork
You've already forked language
 0
Code Issues 139 Pull requests 2 Projects Releases 18 Packages 3 Wiki Activity Actions

test: honour test { coverageThreshold } (coverage measurement) #191

New issue
Closed
opened 2026-10-08 12:44:50 +00:00 by orion · 10 comments
orion commented 2026-10-08 12:44:50 +00:00
Collaborator
Copy link

Context

test { coverageThreshold } is part of the project.trb vocabulary (Int, 0..100, default 0) and is read by nobody: torb test measures no coverage at all. Split out of #33 (#33), because no design record says what coverage is, how it is measured or how a threshold fails a run.

Source

Acceptance criteria

Drafted after the owner decision (see the comments below); the shape depends on q1.

  • The decision is recorded in a design record (PROJECT.md section on test, or a new COVERAGE.md) before code.
  • torb test fails (exit code, message naming measured and required percentage) when measured coverage is below a non-zero coverageThreshold; it prints nothing new and fails nothing when the threshold is 0.
  • Tests for the measurement and for the failure.
  • PROJECT.md, docs/tooling/torb-test.md, docs/tooling/project-trb.md updated; skills regenerated.

Dependencies

Blocked by the owner decision q1 (comments below).

## Context `test { coverageThreshold }` is part of the `project.trb` vocabulary (`Int`, `0..100`, default `0`) and is read by nobody: `torb test` measures no coverage at all. Split out of #33 (https://git.torb.dev/torbscript/language/issues/33), because no design record says what coverage is, how it is measured or how a threshold fails a run. ## Source - [PROJECT.md, the setting's row](https://git.torb.dev/torbscript/language/src/commit/4c6bbc403d419bc769ed3ed387d3d25e629d45ba/docs/design/PROJECT.md#L1093) - [PROJECT.md, what is left](https://git.torb.dev/torbscript/language/src/commit/4c6bbc403d419bc769ed3ed387d3d25e629d45ba/docs/design/PROJECT.md#L1337) - [torb-test.md, Coverage](https://git.torb.dev/torbscript/language/src/commit/4c6bbc403d419bc769ed3ed387d3d25e629d45ba/docs/tooling/torb-test.md#L238) ## Acceptance criteria Drafted after the owner decision (see the comments below); the shape depends on q1. - [ ] The decision is recorded in a design record (PROJECT.md section on `test`, or a new COVERAGE.md) before code. - [ ] `torb test` fails (exit code, message naming measured and required percentage) when measured coverage is below a non-zero `coverageThreshold`; it prints nothing new and fails nothing when the threshold is `0`. - [ ] Tests for the measurement and for the failure. - [ ] PROJECT.md, `docs/tooling/torb-test.md`, `docs/tooling/project-trb.md` updated; skills regenerated. ## Dependencies Blocked by the owner decision q1 (comments below).
orion commented 2026-10-08 12:45:05 +00:00
Author
Collaborator
Copy link

[loop:torbscript] owner decision: what test { coverageThreshold } measures and how a run fails below it (split from #33; the result is a public behaviour of torb test).

What is settled. The setting exists: Int, 0..100, default 0, a problem outside the range (PROJECT.md); it is computed or literal like any non-static setting; torb test reads it nowhere (torb-test.md, Coverage). torb test runs in the VM by default and natively with --native (dev profile); the VM has a line table of statements for the debugger (DEBUGGER.md).

What is not settled (no design record, no CONCEPT entry; nothing in the repo defines "coverage"):

  1. The unit: lines of TorbScript source, functions, or branches.
  2. The mechanism: the VM counting executed statements, or gcov-style instrumentation of the generated C (needs #line mapping back to .trb, which BACKEND 6.3 lists as not done, and GCC/Clang only).
  3. The scope: the package's own src/ only, or also tests, std/, dependencies; per workspace member or aggregated.
  4. When it is measured: always when the threshold is above 0, or behind a flag; what --native does with a non-zero threshold.
  5. The failure: exit code, message, whether a per-file table is printed.

Options.

  • A (recommended). (1) Line coverage: a source line counts when it carries an executable statement. (2) Measured in the VM from the debugger's statement table, so it works on every platform and with every C compiler; the VM stays the default of torb test. (3) Scope: the src/ of the package under test; tests, std/ and dependencies excluded; a workspace member is measured and checked against its own manifest's threshold. (4) Measured only when the threshold is above 0 (no new flag); torb test --native with a non-zero threshold is an error that says coverage is measured in the VM (never silently ignored). (5) Below the threshold the run ends with a non-zero exit code after the normal report and one line coverage 71% is below the threshold 80%; --report json carries a coverage event; a per-file table is a later issue.
  • B. As A, but gcov instrumentation of the native dev binary: matches --native runs and real optimized code paths, but needs #line first, gcov tooling on the machine, no MSVC, and costs a larger slice.
  • C. Function coverage instead of lines (cheaper, coarser; weaker as a gate).
  • D. Decide per slice; no code until 1-5 are answered.

Split after the answer (dependency-linked, label loop): (i) the measurement in the VM with tests; (ii) threshold check, exit code and report line; (iii) docs. Whether the repository's own manifests ever set a threshold is Torben's call, not loop work.

Recommendation. Option A. Say "go A", "go B", "go C" or name the changes.

[loop:torbscript] owner decision: what `test { coverageThreshold }` measures and how a run fails below it (split from #33; the result is a public behaviour of `torb test`). **What is settled.** The setting exists: `Int`, `0..100`, default `0`, a problem outside the range ([PROJECT.md](https://git.torb.dev/torbscript/language/src/commit/4c6bbc403d419bc769ed3ed387d3d25e629d45ba/docs/design/PROJECT.md#L1093)); it is computed or literal like any non-static setting; `torb test` reads it nowhere ([torb-test.md, Coverage](https://git.torb.dev/torbscript/language/src/commit/4c6bbc403d419bc769ed3ed387d3d25e629d45ba/docs/tooling/torb-test.md#L238)). `torb test` runs in the VM by default and natively with `--native` (`dev` profile); the VM has a line table of statements for the debugger ([DEBUGGER.md](https://git.torb.dev/torbscript/language/src/commit/4c6bbc403d419bc769ed3ed387d3d25e629d45ba/docs/design/DEBUGGER.md)). **What is not settled** (no design record, no CONCEPT entry; nothing in the repo defines "coverage"): 1. *The unit:* lines of TorbScript source, functions, or branches. 2. *The mechanism:* the VM counting executed statements, or gcov-style instrumentation of the generated C (needs `#line` mapping back to `.trb`, which BACKEND 6.3 lists as not done, and GCC/Clang only). 3. *The scope:* the package's own `src/` only, or also tests, `std/`, dependencies; per workspace member or aggregated. 4. *When it is measured:* always when the threshold is above 0, or behind a flag; what `--native` does with a non-zero threshold. 5. *The failure:* exit code, message, whether a per-file table is printed. **Options.** - **A (recommended).** (1) Line coverage: a source line counts when it carries an executable statement. (2) Measured in the VM from the debugger's statement table, so it works on every platform and with every C compiler; the VM stays the default of `torb test`. (3) Scope: the `src/` of the package under test; tests, `std/` and dependencies excluded; a workspace member is measured and checked against its own manifest's threshold. (4) Measured only when the threshold is above 0 (no new flag); `torb test --native` with a non-zero threshold is an error that says coverage is measured in the VM (never silently ignored). (5) Below the threshold the run ends with a non-zero exit code after the normal report and one line `coverage 71% is below the threshold 80%`; `--report json` carries a `coverage` event; a per-file table is a later issue. - **B.** As A, but gcov instrumentation of the native `dev` binary: matches `--native` runs and real optimized code paths, but needs `#line` first, gcov tooling on the machine, no MSVC, and costs a larger slice. - **C.** Function coverage instead of lines (cheaper, coarser; weaker as a gate). - **D.** Decide per slice; no code until 1-5 are answered. **Split after the answer** (dependency-linked, label `loop`): (i) the measurement in the VM with tests; (ii) threshold check, exit code and report line; (iii) docs. Whether the repository's own manifests ever set a threshold is Torben's call, not loop work. **Recommendation.** Option A. Say "go A", "go B", "go C" or name the changes.
orion commented 2026-10-08 12:45:05 +00:00
Author
Collaborator
Copy link

Question q1 for torben (decision, blocking): What should test { coverageThreshold } measure, and how does a run fail below it?

v: 1
id: q1
to: torben
kind: decision
blocking: true
question: "What should `test { coverageThreshold }` measure, and how does a run fail below it? Details in the owner decision comment above (unit, mechanism, scope, when measured, failure)."
options:
  - id: a
    label: "A: line coverage counted in the VM"
    detail: Statement lines of the package's src/ from the debugger's table; measured when the threshold is above 0; --native with a threshold is an error; non-zero exit and one message line below it
  - id: b
    label: "B: gcov instrumentation of the native binary"
    detail: Needs #line mapping first, gcov tooling, GCC/Clang only; larger slice
  - id: c
    label: "C: function coverage"
    detail: Cheaper and coarser than lines; weaker as a gate
  - id: d
    label: "D: decide per slice"
    detail: No code until every point is answered
recommendation: a
asked_by: orion
**Question q1** for torben (decision, blocking): What should `test { coverageThreshold }` measure, and how does a run fail below it? ```hos-question v: 1 id: q1 to: torben kind: decision blocking: true question: "What should `test { coverageThreshold }` measure, and how does a run fail below it? Details in the owner decision comment above (unit, mechanism, scope, when measured, failure)." options: - id: a label: "A: line coverage counted in the VM" detail: Statement lines of the package's src/ from the debugger's table; measured when the threshold is above 0; --native with a threshold is an error; non-zero exit and one message line below it - id: b label: "B: gcov instrumentation of the native binary" detail: Needs #line mapping first, gcov tooling, GCC/Clang only; larger slice - id: c label: "C: function coverage" detail: Cheaper and coarser than lines; weaker as a gate - id: d label: "D: decide per slice" detail: No code until every point is answered recommendation: a asked_by: orion ```
orion commented 2026-10-08 12:45:05 +00:00
Author
Collaborator
Copy link

[loop:torbscript] refine: not ready, split from #33 (#33); nothing in the design records defines coverage (unit, mechanism, scope, failure), so the owner decision above and the question q1 below are filed. Seed impact: none expected. Size once decided: two or three issues (measurement, threshold check, docs). next: waiting (owner decision q1)

[loop:torbscript] refine: not ready, split from #33 (https://git.torb.dev/torbscript/language/issues/33); nothing in the design records defines coverage (unit, mechanism, scope, failure), so the owner decision above and the question q1 below are filed. Seed impact: none expected. Size once decided: two or three issues (measurement, threshold check, docs). next: waiting (owner decision q1)
orion commented 2026-10-08 14:36:21 +00:00
Author
Collaborator
Copy link

[loop:torbscript] owner decision recorded (torben via WhatsApp, 2026-10-08 16:35 Europe/Berlin): "Ja, Zeilenabdeckung bei Tests" (yes, line coverage for tests).

Resolution: option A, line coverage counted in the VM over the package's src/ statement lines (debugger table), measured when the threshold is above 0, --native with a threshold is an error, below the threshold the run exits non-zero with one message line. Labelled ready.

[loop:torbscript] owner decision recorded (torben via WhatsApp, 2026-10-08 16:35 Europe/Berlin): "Ja, Zeilenabdeckung bei Tests" (yes, line coverage for tests). Resolution: option A, line coverage counted in the VM over the package's `src/` statement lines (debugger table), measured when the threshold is above 0, `--native` with a threshold is an error, below the threshold the run exits non-zero with one message line. Labelled `ready`.
orion commented 2026-10-08 17:47:14 +00:00
Author
Collaborator
Copy link

[question-closed] q=q1 reason=answered-elsewhere

[question-closed] q=q1 reason=answered-elsewhere
orion commented 2026-10-08 21:04:37 +00:00
Author
Collaborator
Copy link

[loop:torbscript] claim implement by session:loop-torbscript-20261008 until 2026-10-09T03:04Z

[loop:torbscript] claim implement by session:loop-torbscript-20261008 until 2026-10-09T03:04Z
orion commented 2026-10-08 22:03:48 +00:00
Author
Collaborator
Copy link

[loop:torbscript] implement: PR #222, CI success, next: review (waiting on the merge gate argus)

This PR is the first slice (the measurement, Refs #191), since the whole issue does not fit one PR: the threshold check in torb test would collide with examples/game-engine/project.trb (coverageThreshold = 80) and the native gate, which needs a decision of its own. Follow-ups: #230 (wire the check into torb test: message, exit code, JSON event, --native error, the game-engine manifest) and #231 (user docs and skills).

Worker usage: one implement subagent, no further workers.

[loop:torbscript] implement: PR https://git.torb.dev/torbscript/language/pulls/222, CI success, next: review (waiting on the merge gate argus) This PR is the first slice (the measurement, `Refs #191`), since the whole issue does not fit one PR: the threshold check in `torb test` would collide with `examples/game-engine/project.trb` (`coverageThreshold = 80`) and the native gate, which needs a decision of its own. Follow-ups: #230 (wire the check into `torb test`: message, exit code, JSON event, `--native` error, the game-engine manifest) and #231 (user docs and skills). Worker usage: one implement subagent, no further workers.
orion commented 2026-10-08 23:04:33 +00:00
Author
Collaborator
Copy link

[loop:torbscript] implement: PR 222 (measurement slice) merged by argus. The rest of this issue is split into #230 (wire the check into torb test) and #231 (docs); this issue closes when both landed. next: waiting (follow-ups #230 and #231)

[loop:torbscript] implement: PR 222 (measurement slice) merged by argus. The rest of this issue is split into #230 (wire the check into `torb test`) and #231 (docs); this issue closes when both landed. next: waiting (follow-ups #230 and #231)
orion commented 2026-10-09 11:08:12 +00:00
Author
Collaborator
Copy link

[loop:torbscript] implement: measurement slice landed; the rest is #230 (implementation starts now) and #231. next: waiting (#230, #231) until 2026-10-10T04:00:00Z

[loop:torbscript] implement: measurement slice landed; the rest is #230 (implementation starts now) and #231. next: waiting (#230, #231) until 2026-10-10T04:00:00Z
orion commented 2026-10-09 17:00:11 +00:00
Author
Collaborator
Copy link

[loop:torbscript] close: all slices landed - measurement (earlier), #230 threshold check (PR #265) and #231 docs (PR #267), each merged by argus; main CI green (latest checked on d87348ed). Parent closed.

[loop:torbscript] close: all slices landed - measurement (earlier), #230 threshold check (PR https://git.torb.dev/torbscript/language/pulls/265) and #231 docs (PR https://git.torb.dev/torbscript/language/pulls/267), each merged by argus; main CI green (latest checked on d87348ed). Parent closed.
orion closed this issue 2026-10-09 17:00:11 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
loop/705-default-shadow
loop/691-tensor-core
loop/705-fix-candidate
orion/456-tierb-probe
loop/164-measure2
loop/207-measure
loop/164-prototype
probe/207-count
loop/205-measure
loop/140-hide-staged
nightly-20261103
nightly-20261031
nightly-20261029
nightly-20261027
nightly-20261026
nightly-20261025
nightly-20261024
nightly-20261023
nightly-20261021
nightly-20261020
nightly-20261015
nightly-20261014
nightly-20261013
nightly-20261011
nightly-20261010
nightly-20261009
nightly-20261008
seeds
Labels
Clear labels   
loop
Backlog item of the project loop (Orion), see CONTRIBUTING

  
ready
Refined to the definition of ready

No labels
loop
ready
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
torbscript/language#191
No description provided.