perf: torb run --vm of benchmarks/list-iterate is 3x slower with the counted for pass (follow-up of #470) #482
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Context
PR #475 (#470) measured in forge CI run 1745 (
tools/collection-cost.sh, tier B, same job, fastest of 3 runs):torb run --vm benchmarks/list-iterate.trbtook 20 828 ms with the compiler of the merge base (44a359160d) and 62 526 ms with the branch (51d38d87c7).list-indexin the same job: 23 215 ms vs 23 513 ms. The native numbers improved (10.61x to 1.47x), the criterion of #470 ("torb runoflist-iterate(VM) must not get slower") was not met.Second sample (refine, 2026-10-27): the tier B log of the run for the PR's second head (run 1799, job 20305) prints base 20 339 ms and head 56 874 ms for
list-iterate(2.8x),list-index23 368 ms vs 22 741 ms. The 3x reproduces; it is not noise. #475 is merged (22cf1658) somaincarries it; the guard oftools/collection-cost.shdoes not fire on amainpush (no merge-base diff), so the next sample must come from a PR (see Measurement).Goal
Find where the VM spends the time of the counted loop of
list-iterate(less,branch, aListItemAddressplace read andadd.i64.unchecked; candidates: the new slots of the pass not known to the bytecode emitter's register/place tables,placeOfincompiler/src/backend/bytecode/emit.trbfor thePlaceStep.ListItemstep (the uncheckedElementstep the pass forms), theReadof the cursor'sitemsfield, or the VM'sListItemAddresskernel call incompiler/src/vm/interpret.trb), then fix it in the emitter or the VM, not in the pass.Measurement (CI only, first step of the implementer)
tools/collection-cost.shis the instrument: it builds the merge-base compiler and this checkout's in one job and printsVM, fastest of 3: list-index X ms, list-iterate Y msper compiler. Its guard measures only when the branch diff touchesstd/collections,std/core/src/buffer.trb,runtime,compiler/src/irorbenchmarks; a fix incompiler/src/backend/bytecodeorcompiler/src/vmwould not trigger it. So commit 1 of the PR widens the guard bycompiler/src/backend/bytecodeandcompiler/src/vm, and makes the script also print, for base and head, the bytecode of the benchmark (torb ir --bytecode benchmarks/list-iterate.trb, loop blocks only) so the job log shows what the pass changed in the loop. Push it, read tier B of that CI run (GET /repos/torbscript/language/actions/runs/<run>/jobs,/actions/jobs/<id>/logs, seeci-wait.sh), name the root cause from the bytecode diff and the VM line (base vs head with identical emitter/VM is the noise floor), then commit 2 fixes it.Acceptance criteria
collection-cost.sh, same job):list-iterateVM head <= 1.10 x base (base was 20 339-20 828 ms in two samples, so head <= ~22 400 ms; was 56 874-62 526 ms). Two samples recorded in the PR body (each CI run of the PR is one).list-indexVM head <= 1.10 x base in the same table (was 0.97-1.02x).list-iteratehead <= 1.7x (was 1.47x-1.59x), allocations oflist-index/list-iterateunchanged (22 live at exit, 24 total as in #475).compiler/tests, or a conformance program) so a regression shows in tier A.sh tools/gates.sh agreen in the PR's CI.Design / seed / size
docs/PERFORMANCE.mdsection 5.1 (what P11 builds) and finding F5; ROADMAP: performance round P11 follow-up. The pass (ir/counted.trb) stays as is.chore(tools): measure the bytecode and the VM in collection-cost.shandperf(bytecode): <root cause>(orperf(vm): ...).Out of scope
The native back end (done in #475), the counted pass itself, seeds.
[loop:torbscript] claim refine by session:loop-torbscript-20261027 until 2026-10-27T01:13Z
[loop:torbscript] decided by the loop: fix in the emitter or VM, not in the pass, and measure in CI first. Numbers: the 3x reproduces (run 1745: 20 828 ms base vs 62 526 ms head; run 1799: 20 339 vs 56 874 ms;
list-indexflat), so no owner question.collection-cost.shdoes not measure a change undercompiler/src/backend/bytecodeorcompiler/src/vm, so the first commit widens its guard and prints the bytecode of base and head; the root cause is read from that CI log. Seed untouched.[loop:torbscript] refine: #475 is merged (
22cf1658); the 3x reproduced in a second CI sample (VMlist-iterate20 339 ms base vs 56 874 ms head, run 1799; first 20 828 vs 62 526). Issue body now holds goal, CI measurement step (widen thecollection-cost.shguard to bytecode/vm and print the bytecode of base and head, read tier B), criteria with numbers (VMlist-iteratehead <= 1.10 x base ~ 22 400 ms,list-index<= 1.10x, native 1.47x-1.59x and allocations unchanged), design point PERFORMANCE.md 5.1/F5, seed impact no, size 1 PR / 2 commits. Labelreadyset. next: implement[loop:torbscript] claim implement by session:loop-torbscript-20261027 until 2026-10-27T05:37Z
[loop:torbscript] checkpoint implement: commit 1 pushed on loop/482-vm-list-iterate (worktree code/wt/torbscript-language-482), draft PR open; waiting for tier B of its CI to read the base/head bytecode and VM line. next: implement (commit 2 = fix)
[loop:torbscript] claim implement by session:loop-torbscript-20261027 until 2026-10-27T06:35Z
[loop:torbscript] claim implement by session:loop-torbscript-20261027 until 2026-10-27T06:58Z
[loop:torbscript] implement: checkpoint 1, not finished. Commit 1 (
chore(tools): measure the bytecode and the VM in collection-cost.sh, now8cd52b0fafter an amend to re-trigger CI) is onloop/482-vm-list-iterate, draft PR #520. The CI of its first head (run 1982) sat inwaitingfor 25+ min without a runner (all jobs, evencommit messages; the CI queue held 5 further waiting runs of other PRs), so I amended the commit (same content) and force-pushed with lease; run 1987 is queued the same way, tier B has not started, no base/head numbers or bytecode yet. Candidates stay as in the issue (emitterPlaceStep.ListItematemit.trb:1620, VMlistItemStep->kernelCall ... ListItemAddressinresolvePlace,interpret.trb:1795); I decide only with the tier B log, no fix written blind.Remaining steps: (1) wait for run 1987 (
ci-wait.sh -C code/wt/torbscript-language-482), read tier B job log:VM, fastest of 3line and the two bytecode dumps oflist-iterate; (2) name the root cause from the bytecode diff; (3) commit 2perf(bytecode|vm): <root cause>with a test pinning it undercompiler/tests; (4) push, read tier A/B (criteria: VMlist-iteratehead <= 1.10 x base,list-index<= 1.10x, native <= 1.7x, allocations 22/24), put two samples and the root cause in the PR body, mark the PR ready; (5)tools/check-commits.sh.CONTINUE: torbscript/language#482
next: waiting (CI) until 2026-10-27T02:30:00Z
[loop:torbscript] release implement: checkpoint 1 pushed, CI queue (no runner for the PR run); the claim is released, the next tick continues at step (1) above.
ArrayListis a type overBuffer(step 3 of #55, commit 2 of 2) #526ArrayListis a type overBuffer(step 3 of #55, commit 2 of 2) #526[loop:torbscript] claim implement (resume at checkpoint 1, PR #520) by session:loop-torbscript-20261027 until 2026-10-27T09:15Z
ArrayListis a type overBuffer(step 3 of #55, commit 2 of 2) #526[loop:torbscript] implement: checkpoint 2, not finished. Tier B of PR #520 (run 1987, head
8cd52b0f, job log read; the merge baseb81b31f4already carries #475, so base and head have the same emitter and VM):VM, fastest of 3base list-index 22 672 ms / list-iterate 62 358 ms, head 23 177 / 61 511 ms (noise floor confirmed; native rows 1.48x/1.58x, allocations 22/22 unchanged). The bytecode of the new loop is minimal (less,branch,load r9, place 1with theListItemstep,add.i64.uncheckedx2,add.i64), so the emitter is not at fault.Root cause (from the code, to be confirmed by the fix's CI run):
kernelCall(compiler/src/vm/interpret.trb:1670) turns theKernelOperationinto its number withkernelNumberOf(compiler/src/backend/bytecode/format.trb:917), a linear scan overallKernelOperationscomparing each candidate.ElementAddress(whatlist-indexuses) sits near the front of the list (position ~13),ListItemAddress(operation 68, added late) near the back, so every item read of the counted loop scans ~68 entries before the kernel runs; ~500 ns per item matches the 80M reads (62 s - 20 s).Remaining steps: (1) commit 2
perf(vm): look up a kernel operation's number without scanning the list: make the number O(1) (e.g. amatchinkernelNumberOf, or the number cached for the place steps inresolvePlace), keep the table and the CTORB_OPERATION_*numbering identical; (2) a test pinning it undercompiler/tests(everyKernelOperationinallKernelOperationsorder maps to its index, so a table change that breaks the lookup shows in tier A); (3) push, read tier B (criteria: VMlist-iteratehead <= ~22 400 ms against the 20 339-20 828 ms pre-#475 baseline from the issue, since base now equals head;list-index<= 1.10x, native <= 1.7x, allocations 22/22); (4) PR body: root cause, two samples (run 1987 above is the "before" one), mark ready; (5)tools/check-commits.sh. Note for the PR body:tools/collection-cost.shcompares to the merge base, so after #475 its VM base forlist-iterateis the slow one; the pre-#475 numbers come from the issue.Worker usage: ~100k tokens of context (read-only analysis, no code written).
CONTINUE: torbscript/language#482
next: implement (commit 2)
[loop:torbscript] claim implement (continue from the last checkpoint) by session:loop-torbscript-20261027 until 2026-10-27T09:30Z
[loop:torbscript] implement: checkpoint 3, code written and pushed, CI queued, not finished. Commit 2
perf(vm): look up a kernel operation's number without scanning the list(00a63378, rebased on main142293fd,check-commits.shclean) is onloop/482-vm-list-iterate:kernelNumberOfis amatchgenerated fromallKernelOperations(91 operations, same order, so the C numbering is unchanged);bytecode.test.trbalready pinned every operation to its table index, and gets one more test (last operation belowkernelNativeBase). PR #520 body is updated (root cause, fix, the run 1987 "before" sample); still a draft.CI: the forge runner pool is saturated (runs 1995, 1998 running, 2000, 2002 ahead of ours); this head's runs 2004 and 2005 (PR #520) were still
waitingafter 50 min ofci-wait.sh, nothing of mine ran. Not touchingtools/collection-cost.shfurther (another worker owns #526).Remaining: (1)
ci-wait.shon00a63378until runs 2004/2005 finish; read tier A and tier B (criteria: VMlist-iteratehead <= ~22 400 ms,list-index<= 1.10x, native <= 1.7x, allocations 22/22; a miss means the scan was not the (whole) cause: then bytecode ofListItemAddresspath inkernelCall, e.g. thekernellist refill per call); (2) add the "after" sample to the PR body, mark #520 ready (not draft); (3) wait on the merge gate.Worker usage: ~95k tokens of context.
CONTINUE: torbscript/language#482
next: implement (read CI of
00a63378, tier B numbers into the PR body, mark ready)[loop:torbscript] claim implement (resume at checkpoint 3, PR #520) by session:loop-torbscript-20261027-0718 until 2026-10-27T12:31Z
[loop:torbscript] implement: done, PR #520 is ready and Argus is requested; the claim is released.
Rebased onto the current
main(2714b259, linear,tools/check-commits.shclean, force-push toloop/482-vm-list-iterateonly); head is now24803ba8(same two commits as00a63378, only the base moved), the PR is mergeable. CI on24803ba8was queued when I handed over (runs 2029, 2030); the rebase brought only unrelated release/deploy changes.Tier B of the previous head
00a63378(run 2004, forge CI green incl. tier A and gates),VM, fastest of 3:list-iterate62 321 ms (base) -> 12 143 ms (criterion <= ~22 400 ms; pre-#475 baseline 20 339-20 828 ms), so the O(1)kernelNumberOffixes the regression and beats the old figure.list-index23 217 -> 23 476 ms (1.011x, criterion <= 1.10x).list-iterate1.60x, criterion <= 1.7x), allocations 22/22, hello-world size +0 bytes.The "after" sample is in the PR body, WIP/draft marking is dropped from the title, review requested from
argus.Worker usage: ~60k tokens of context (rebase, CI log read, PR update).
next: review