perf: an allocation budget per conformance program as a gate #60

New issue

Closed

opened 2026-10-05 21:07:15 +00:00 by orion · 7 comments

orion commented 2026-10-05 21:07:15 +00:00

Collaborator

Copy link

Context

Make the allocation count of each conformance program a gate so a regression is noticed.

Source

• PERFORMANCE.md section 6

Acceptance criteria

• Each program's allocations recorded
• A gate fails on growth beyond the budget

Dependencies

None known.

Moved from the planning lists of the docs; the design record stays the specification.

## Context Make the allocation count of each conformance program a gate so a regression is noticed. ## Source - [PERFORMANCE.md section 6](https://git.torb.dev/torbscript/language/src/commit/393ef2608e4a60b8ff76d8b6b408951b822d887d/docs/PERFORMANCE.md#L1264) ## Acceptance criteria - [ ] Each program's allocations recorded - [ ] A gate fails on growth beyond the budget ## Dependencies None known. _Moved from the planning lists of the docs; the design record stays the specification._

orion added this to the Kernel of natives milestone 2026-10-05 21:07:15 +00:00

orion added the

loop

label 2026-10-05 21:07:15 +00:00

orion commented 2026-10-09 05:18:19 +00:00

Author

Collaborator

Copy link

[loop:torbscript] intake: in scope (author orion, label loop), source PERFORMANCE.md section 6, not a duplicate, no collision with a design record. A tier B or conformance gate with a recorded budget per program; measurable, so the loop decides the budget rule from the numbers in refine. next: refine, waiting (ready buffer full: #4, #163, #164, #223 are ready) until 2026-10-10T04:00:00Z

[loop:torbscript] intake: in scope (author orion, label loop), source PERFORMANCE.md section 6, not a duplicate, no collision with a design record. A tier B or conformance gate with a recorded budget per program; measurable, so the loop decides the budget rule from the numbers in refine. next: refine, waiting (ready buffer full: #4, #163, #164, #223 are ready) until 2026-10-10T04:00:00Z

orion commented 2026-10-10 00:44:18 +00:00

Author

Collaborator

Copy link

[loop:torbscript] claim refine by session:loop-torbscript-20261010 until 2026-10-10T02:45Z (the "ready buffer full" marker is stale: only seed-blocked items are ready)

[loop:torbscript] claim refine by session:loop-torbscript-20261010 until 2026-10-10T02:45Z (the "ready buffer full" marker is stale: only seed-blocked items are ready)

orion commented 2026-10-10 00:46:50 +00:00

Author

Collaborator

Copy link

[loop:torbscript] refine: ready, an exact per-program allocation budget, one PR in two commits, no seed impact.

Goal. tests/conformance/ already runs each program with TORB_REPORT_LEAKS=1 and asserts live blocks at exit: 0 (tools/conformance.sh, the leak run). Add a second number to that same report, the total blocks the program allocated, and hold each program to a recorded budget, so a regression from per element to per loop allocation fails tier B instead of being found later. Implements docs/PERFORMANCE.md section 6, "An allocation budget per conformance program"; the design record stays that section.

Acceptance criteria.

1. runtime/memory.c: every block made by torb_allocate (and so torb_allocate_zeroed), torb_raw_allocate (and so torb_raw_allocate_zeroed) and the immortal path is counted once into a new per-heap counter (torb_heap.allocated_blocks plus machine_allocated_blocks, next to the live and immortal ones in runtime/include/torb_pool.h). torb_raw_count_immortal and torb_raw_count_mortal re-class a block and do not count. torb_fold_worker in runtime/task.c folds the counter into worker 0 and a torb_pool_sum_* pair sums it, so a program with workers loses nothing; a new torb_allocated_block_count() sits beside torb_live_block_count().
2. torb_report_leaks prints a third line allocated blocks at exit: N, in the VM branch (machine_allocated_blocks) and the native one. The two existing lines and the greps on them are unchanged.
3. runtime/tests/memory_test.c: a test that the counter rises by exactly the number of allocations, is not lowered by a free, and counts an immortal block once.
4. tools/conformance.sh (native leg): programs that get the leak run today (no .stderr, no .leaks, not binary-only/) are also held to <program>.allocations. The file holds one integer, or unbudgeted: <reason> (as .leaks holds its reason, and for the same kind of reason). The count is read from a run with TORB_WORKERS=1; a program with a .workers file gets that one extra run, since its leak run uses its own worker count. The program fails when the count is above its budget, printing program, budget, count and the difference. A count below the budget passes and is listed in a closing note with the command to lower it. A program with a leak run and no .allocations file fails and prints its count (this also makes every new program carry a budget).
5. sh tools/conformance.sh --update-budgets [--filter x] writes .allocations from a run. It reads the count three times and writes nothing for a program whose three differ, naming it, so it is unbudgeted by hand. --update leaves budgets alone: raising one is a separate, visible act, and the commit message says why.
6. The VM legs (--vm, TORB_VM_DEBUGGED=1) print nothing new and read no budget. Whether the VM allocates like the native binary is not measured yet; that is a follow-up once the budgets exist.
7. Every program of the suite ends the PR with a .allocations file (a number, or unbudgeted: with a reason, expected for timing-driven programs such as process-pipes, confined-child, task-sleep, entropy, wall-clock, io-eight-files and the like if their count varies); the final tip of the branch is green in forge tier B.
8. Docs: tests/conformance/README.md (the table of what is compared gets the .allocations row, the "Zero live blocks" section the third line, and how to update); compiler/CONTRIBUTING.md line "Numbers are recorded, not enforced: allocation counts, ..." is corrected, since the conformance allocation count now does fail a gate (the benchmark ratios and the size of the C stay recorded); docs/PERFORMANCE.md section 6 paragraph states what was built and the rule below, and the status line is updated. Docs changes regenerate skills/ (docs skill); the worker does it with the existing build/release/torb if present, else the first CI run names the diff.

Decided by the loop (budget rule).

• Recorded as: one sidecar file per program, next to .expected, because a loop and its parallel PRs add programs at the same time and a single list file would conflict on every merge. Reviewers see a budget change as a one-line diff in the PR.
• Tolerance: none (exact, growth fails). With one worker, a fixed profile and no timing in the program, the number of runtime blocks is a function of the emitted C, not of the machine or the C compiler (a counter call has a side effect and cannot be elided). docs/PERFORMANCE.md already reads this column as "exact" (sections 4 and 6; for example list-iterate 24 blocks, pipeline 85, map-count 50 110). A percentage tolerance would let a per-loop block creep in unseen on the small programs where it matters (24 blocks), and an exact number makes the one-block change of a std or lowering round show up in review. The only exceptions are programs whose count varies between runs; they take unbudgeted: with a reason.
• Shrinking is allowed without touching the file. An improvement passes; the note tells the author to lower the budget so the gain is kept (a ratchet by convention, not a red gate, so unrelated rounds are not blocked by someone else's improvement).
• Where it runs: tier B, native conformance leg, on the forge's tier B (linux-x64) job (sh tools/gates.sh b). It is not a tier A gate (it needs a built binary and the C compiler). The runner enforces budgets on Linux x86_64 and only reports (prints the count, compares nothing) elsewhere, via uname, with TORB_ALLOCATION_BUDGET=enforce|report to override: .forgejo/actions/portable and the GitHub mirror run the same tools/conformance.sh on windows-x64, macos-arm64 and linux-arm64, and the counted raw blocks of the OS layer (console.c wide buffers, file.c path conversions, process.c) are not the same on every platform. Counting the raw blocks at all is deliberate: the map's entry table and bucket array (map.c) are raw blocks and are exactly what a per-element regression of a map would allocate.
• Numbers: the counter does not exist yet, so no CI log has them. Per the contract they are measured in forge CI first: the second commit is pushed without the .allocations files, its tier B run fails every program with no budget, allocated N, the implementer writes the files from that log (a script that asserts each name matches), and the second run with the files must be green. That second run is also the stability check, and a program that differs between the two runs becomes unbudgeted:. (Nothing runs the suite in the worker's own exec: orion/infra#947.)

Seed impact: none. No syntax change, no new native (the counter is C-internal and no TorbScript native is added), no driver flag, no std name looked up by string. The runtime change is read by the bootstrap's own binary and by the conformance runner of the same checkout; the seed's program.c is untouched. Do not touch seed/.

Size: one PR, two Conventional Commits.

1. feat(runtime): count the blocks a program allocates - runtime/memory.c, runtime/include/torb_pool.h, runtime/include/torb.h, runtime/task.c, runtime/tests/memory_test.c, the leak-report comment in runtime/process.c; closes the gap for workers (torb_fold_worker).
2. test(conformance): hold each program to an allocation budget - tools/conformance.sh, the .allocations files, tests/conformance/README.md, compiler/CONTRIBUTING.md, docs/PERFORMANCE.md, skills/. Implementer: opus/high for the runtime (the contract's rule for runtime/); the second commit is tooling plus data. The ~280 data files are one mechanical change of one commit, not a reason to split.

Overlap with open PRs (no blocker, order only).

• #291 (conformance in the debugged VM) edits tools/conformance.sh (header comment, the timings file selection near the end), tools/gates.sh, compiler/CONTRIBUTING.md (tier B paragraph a few lines above the "Numbers are recorded" sentence and the timings paragraph), docs/design/DEBUGGER.md, compiler/src/ir/lower/scope.trb. This issue's second commit touches the same two files in different hunks; implement on top of #291 if it has landed, otherwise expect a trivial rebase in conformance.sh's header and in CONTRIBUTING.md. The debugged leg is a --vm leg, so it reads no budget.
• #284 (perf(ir), dead witness members) edits compiler/src/backend/{bytecode/emit,c/body,c/emit}.trb, compiler/src/ir/{called-members,ir,kept,lower/lower,print,sites,verify}.trb, tests emit-c, lower-generics, lower, witness-dead, docs/BACKEND.md and docs/PERFORMANCE.md (status line, F12 and the P10 row). No file in common except docs/PERFORMANCE.md: section 6 here against sections 5 and F12 there, so a clean rebase is likely. #284 removes emitted table entries, not blocks, so it should not move a budget; if it does, the first budget run of whichever lands second shows it.

[loop:torbscript] refine: ready, an exact per-program allocation budget, one PR in two commits, no seed impact. **Goal.** `tests/conformance/` already runs each program with `TORB_REPORT_LEAKS=1` and asserts `live blocks at exit: 0` (`tools/conformance.sh`, the leak run). Add a second number to that same report, the total blocks the program allocated, and hold each program to a recorded budget, so a regression from per element to per loop allocation fails tier B instead of being found later. Implements `docs/PERFORMANCE.md` section 6, "An allocation budget per conformance program"; the design record stays that section. **Acceptance criteria.** 1. `runtime/memory.c`: every block made by `torb_allocate` (and so `torb_allocate_zeroed`), `torb_raw_allocate` (and so `torb_raw_allocate_zeroed`) and the immortal path is counted once into a new per-heap counter (`torb_heap.allocated_blocks` plus `machine_allocated_blocks`, next to the live and immortal ones in `runtime/include/torb_pool.h`). `torb_raw_count_immortal` and `torb_raw_count_mortal` re-class a block and do not count. `torb_fold_worker` in `runtime/task.c` folds the counter into worker 0 and a `torb_pool_sum_*` pair sums it, so a program with workers loses nothing; a new `torb_allocated_block_count()` sits beside `torb_live_block_count()`. 2. `torb_report_leaks` prints a third line `allocated blocks at exit: N`, in the VM branch (`machine_allocated_blocks`) and the native one. The two existing lines and the greps on them are unchanged. 3. `runtime/tests/memory_test.c`: a test that the counter rises by exactly the number of allocations, is not lowered by a free, and counts an immortal block once. 4. `tools/conformance.sh` (native leg): programs that get the leak run today (no `.stderr`, no `.leaks`, not `binary-only/`) are also held to `<program>.allocations`. The file holds one integer, or `unbudgeted: <reason>` (as `.leaks` holds its reason, and for the same kind of reason). The count is read from a run with `TORB_WORKERS=1`; a program with a `.workers` file gets that one extra run, since its leak run uses its own worker count. The program fails when the count is above its budget, printing program, budget, count and the difference. A count below the budget passes and is listed in a closing note with the command to lower it. A program with a leak run and no `.allocations` file fails and prints its count (this also makes every new program carry a budget). 5. `sh tools/conformance.sh --update-budgets [--filter x]` writes `.allocations` from a run. It reads the count three times and writes nothing for a program whose three differ, naming it, so it is `unbudgeted` by hand. `--update` leaves budgets alone: raising one is a separate, visible act, and the commit message says why. 6. The VM legs (`--vm`, `TORB_VM_DEBUGGED=1`) print nothing new and read no budget. Whether the VM allocates like the native binary is not measured yet; that is a follow-up once the budgets exist. 7. Every program of the suite ends the PR with a `.allocations` file (a number, or `unbudgeted:` with a reason, expected for timing-driven programs such as `process-pipes`, `confined-child`, `task-sleep`, `entropy`, `wall-clock`, `io-eight-files` and the like if their count varies); the final tip of the branch is green in forge tier B. 8. Docs: `tests/conformance/README.md` (the table of what is compared gets the `.allocations` row, the "Zero live blocks" section the third line, and how to update); `compiler/CONTRIBUTING.md` line "Numbers are recorded, not enforced: allocation counts, ..." is corrected, since the conformance allocation count now does fail a gate (the benchmark ratios and the size of the C stay recorded); `docs/PERFORMANCE.md` section 6 paragraph states what was built and the rule below, and the status line is updated. Docs changes regenerate `skills/` (`docs skill`); the worker does it with the existing `build/release/torb` if present, else the first CI run names the diff. **Decided by the loop (budget rule).** - *Recorded as:* one sidecar file per program, next to `.expected`, because a loop and its parallel PRs add programs at the same time and a single list file would conflict on every merge. Reviewers see a budget change as a one-line diff in the PR. - *Tolerance: none (exact, growth fails).* With one worker, a fixed profile and no timing in the program, the number of runtime blocks is a function of the emitted C, not of the machine or the C compiler (a counter call has a side effect and cannot be elided). `docs/PERFORMANCE.md` already reads this column as "exact" (sections 4 and 6; for example `list-iterate` 24 blocks, `pipeline` 85, `map-count` 50 110). A percentage tolerance would let a per-loop block creep in unseen on the small programs where it matters (24 blocks), and an exact number makes the one-block change of a std or lowering round show up in review. The only exceptions are programs whose count varies between runs; they take `unbudgeted:` with a reason. - *Shrinking is allowed without touching the file.* An improvement passes; the note tells the author to lower the budget so the gain is kept (a ratchet by convention, not a red gate, so unrelated rounds are not blocked by someone else's improvement). - *Where it runs:* tier B, native conformance leg, on the forge's `tier B (linux-x64)` job (`sh tools/gates.sh b`). It is not a tier A gate (it needs a built binary and the C compiler). The runner enforces budgets on Linux x86_64 and only reports (prints the count, compares nothing) elsewhere, via `uname`, with `TORB_ALLOCATION_BUDGET=enforce|report` to override: `.forgejo/actions/portable` and the GitHub mirror run the same `tools/conformance.sh` on windows-x64, macos-arm64 and linux-arm64, and the counted raw blocks of the OS layer (`console.c` wide buffers, `file.c` path conversions, `process.c`) are not the same on every platform. Counting the raw blocks at all is deliberate: the map's entry table and bucket array (`map.c`) are raw blocks and are exactly what a per-element regression of a map would allocate. - *Numbers:* the counter does not exist yet, so no CI log has them. Per the contract they are measured in forge CI first: the second commit is pushed without the `.allocations` files, its tier B run fails every program with `no budget, allocated N`, the implementer writes the files from that log (a script that asserts each name matches), and the second run with the files must be green. That second run is also the stability check, and a program that differs between the two runs becomes `unbudgeted:`. (Nothing runs the suite in the worker's own exec: orion/infra#947.) **Seed impact: none.** No syntax change, no new native (the counter is C-internal and no TorbScript `native` is added), no driver flag, no std name looked up by string. The runtime change is read by the bootstrap's own binary and by the conformance runner of the same checkout; the seed's `program.c` is untouched. Do not touch `seed/`. **Size: one PR, two Conventional Commits.** 1. `feat(runtime): count the blocks a program allocates` - `runtime/memory.c`, `runtime/include/torb_pool.h`, `runtime/include/torb.h`, `runtime/task.c`, `runtime/tests/memory_test.c`, the leak-report comment in `runtime/process.c`; closes the gap for workers (`torb_fold_worker`). 2. `test(conformance): hold each program to an allocation budget` - `tools/conformance.sh`, the `.allocations` files, `tests/conformance/README.md`, `compiler/CONTRIBUTING.md`, `docs/PERFORMANCE.md`, `skills/`. Implementer: opus/high for the runtime (the contract's rule for `runtime/`); the second commit is tooling plus data. The ~280 data files are one mechanical change of one commit, not a reason to split. **Overlap with open PRs (no blocker, order only).** - #291 (conformance in the debugged VM) edits `tools/conformance.sh` (header comment, the timings file selection near the end), `tools/gates.sh`, `compiler/CONTRIBUTING.md` (tier B paragraph a few lines above the "Numbers are recorded" sentence and the timings paragraph), `docs/design/DEBUGGER.md`, `compiler/src/ir/lower/scope.trb`. This issue's second commit touches the same two files in different hunks; implement on top of #291 if it has landed, otherwise expect a trivial rebase in `conformance.sh`'s header and in `CONTRIBUTING.md`. The debugged leg is a `--vm` leg, so it reads no budget. - #284 (`perf(ir)`, dead witness members) edits `compiler/src/backend/{bytecode/emit,c/body,c/emit}.trb`, `compiler/src/ir/{called-members,ir,kept,lower/lower,print,sites,verify}.trb`, tests `emit-c`, `lower-generics`, `lower`, `witness-dead`, `docs/BACKEND.md` and `docs/PERFORMANCE.md` (status line, F12 and the P10 row). No file in common except `docs/PERFORMANCE.md`: section 6 here against sections 5 and F12 there, so a clean rebase is likely. #284 removes emitted table entries, not blocks, so it should not move a budget; if it does, the first budget run of whichever lands second shows it.

orion added the

ready

label 2026-10-10 00:46:50 +00:00

orion commented 2026-10-10 00:48:13 +00:00

Author

Collaborator

Copy link

[loop:torbscript] claim implement by session:loop-torbscript-20261010 until 2026-10-10T06:50Z (ready, refine comment above)

[loop:torbscript] claim implement by session:loop-torbscript-20261010 until 2026-10-10T06:50Z (ready, refine comment above)

orion referenced this issue from a commit 2026-10-10 00:54:58 +00:00

feat(runtime): count the blocks a program allocates

orion referenced this issue from a commit 2026-10-10 00:54:58 +00:00

test(conformance): hold each program to an allocation budget

orion referenced this issue from a pull request that will close it, 2026-10-10 00:55:08 +00:00

test(conformance): hold each program to an allocation budget #297

orion referenced this issue from a commit 2026-10-10 01:36:50 +00:00

feat(runtime): count the blocks a program allocates

orion referenced this issue from a commit 2026-10-10 01:36:50 +00:00

test(conformance): hold each program to an allocation budget

orion commented 2026-10-10 08:02:17 +00:00

Author

Collaborator

Copy link

[loop:torbscript] claim implement by session:loop-torbscript-20261010 until 2026-10-10T14:02Z (re-claim: the previous worker died in the gateway gap between 02:10Z and 08:00Z without a write-back)

[loop:torbscript] claim implement by session:loop-torbscript-20261010 until 2026-10-10T14:02Z (re-claim: the previous worker died in the gateway gap between 02:10Z and 08:00Z without a write-back)

orion referenced this issue from a commit 2026-10-10 08:03:54 +00:00

feat(runtime): count the blocks a program allocates

orion referenced this issue from a commit 2026-10-10 08:03:54 +00:00

test(conformance): hold each program to an allocation budget

orion referenced this issue from a commit 2026-10-10 08:05:56 +00:00

feat(runtime): count the blocks a program allocates

orion referenced this issue from a commit 2026-10-10 08:05:56 +00:00

test(conformance): hold each program to an allocation budget

orion referenced this issue from a commit 2026-10-10 08:52:30 +00:00

test(conformance): hold each program to an allocation budget

orion referenced this issue from a commit 2026-10-10 09:36:40 +00:00

feat(runtime): count the blocks a program allocates

orion referenced this issue from a commit 2026-10-10 09:36:40 +00:00

test(conformance): hold each program to an allocation budget

orion referenced this issue from a commit 2026-10-10 10:28:32 +00:00

test(conformance): hold each program to an allocation budget

orion referenced this issue from a commit 2026-10-10 11:02:59 +00:00

feat(runtime): count the blocks a program allocates

orion referenced this issue from a commit 2026-10-10 11:02:59 +00:00

test(conformance): hold each program to an allocation budget

orion referenced this issue from a commit 2026-10-10 11:57:35 +00:00

feat(runtime): count the blocks a program allocates

orion referenced this issue from a commit 2026-10-10 11:57:35 +00:00

test(conformance): hold each program to an allocation budget

orion commented 2026-10-10 13:00:48 +00:00

Author

Collaborator

Copy link

[loop:torbscript] claim implement by session:loop-torbscript-20261010 until 2026-10-10T17:00Z (renewed: resume 3 worker is running on PR 297)

[loop:torbscript] claim implement by session:loop-torbscript-20261010 until 2026-10-10T17:00Z (renewed: resume 3 worker is running on PR 297)

orion referenced this issue from a commit 2026-10-10 13:06:57 +00:00

feat(runtime): count the blocks a program allocates

orion referenced this issue from a commit 2026-10-10 13:06:57 +00:00

test(conformance): hold each program to an allocation budget

argus closed this issue 2026-10-10 17:35:48 +00:00

argus referenced this issue from a commit 2026-10-10 17:35:50 +00:00

feat(runtime): count the blocks a program allocates

argus referenced this issue from a commit 2026-10-10 17:35:50 +00:00

test(conformance): hold each program to an allocation budget

orion commented 2026-10-10 17:36:02 +00:00

Author

Collaborator

Copy link

[loop:torbscript] done: PR #297 merged by argus (b0f6808). Issue closed by the merge; no after-merge steps (no release or deploy). Budgets: 216 exact, 21 unbudgeted:.

[loop:torbscript] done: PR #297 merged by argus (b0f6808). Issue closed by the merge; no after-merge steps (no release or deploy). Budgets: 216 exact, 21 `unbudgeted:`.

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

loop/705-default-shadow

loop/691-tensor-core

loop/705-fix-candidate

orion/456-tierb-probe

loop/164-measure2

loop/207-measure

loop/164-prototype

probe/207-count

loop/205-measure

loop/140-hide-staged

nightly-20261103

nightly-20261031

nightly-20261029

nightly-20261027

nightly-20261026

nightly-20261025

nightly-20261024

nightly-20261023

nightly-20261021

nightly-20261020

nightly-20261015

nightly-20261014

nightly-20261013

nightly-20261011

nightly-20261010

nightly-20261009

nightly-20261008

seeds

Labels

Clear labels   

loop

Backlog item of the project loop (Orion), see CONTRIBUTING

  

ready

Refined to the definition of ready

No labels

loop

ready

Milestone

Clear milestone

No items

No milestone

Kernel of natives

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

torbscript/language#60

No description provided.