perf(runtime): inline the fast path of torb_retain, torb_release and torb_make_unique (P12) #444

Merged

argus merged 3 commits from loop/59-inline-refcount-fastpath into main 2026-10-23 22:35:06 +00:00

Conversation 1 Commits 3 Files changed 6 +189 -38

orion commented 2026-10-23 22:04:37 +00:00

Collaborator

Copy link

Closes #59

What

Round P12 (docs/PERFORMANCE.md finding 13 again): the common case of torb_retain, torb_release and torb_make_unique is static inline in runtime/include/torb.h; every other case is the out-of-line torb_retain_slow, torb_release_slow and torb_make_unique_slow in runtime/memory.c (NULL, immortal, shared/threaded, the last count with the drop depth bound, the zero-count panic, the copy). Semantics of memory.c are unchanged.

Three commits: fix(benchmarks) (run.sh --allocations did not link runtime/os/ and the threads, so the allocation column was empty), perf(runtime), docs(performance). The fix is a prerequisite for the allocation check of this round; it could not be folded into either of the two.

Measurement

One forge CI job, one runner (Linux 6.8, AMD EPYC Genoa, 16 processors, gcc 13.3.0, -O2), benchmarks/run.sh --allocations, RUNS=9, passes A B C D then D C B A, fastest of the eighteen runs. Variants: A as is, B retain inline, C B plus the unique test, D all three. Base: main at ff037c08. The temporary measurement job and script are not part of this branch (they were removed from the history; the job was a push-triggered workflow on this branch's earlier commits).

Program	A us	B us	C us	D us	B	C	D
arithmetic	236 925	237 606	241 680	238 782	+0.3%	+2.0%	+0.8%
call-depth	476 931	481 882	485 981	480 944	+1.0%	+1.9%	+0.8%
wrapper	113 819	114 163	113 909	113 773	+0.3%	+0.1%	-0.0%
closure	198 050	199 480	198 771	197 821	+0.7%	+0.4%	-0.1%
list-index	190 037	192 543	193 971	194 653	+1.3%	+2.1%	+2.4%
list-iterate	468 418	394 924	402 555	346 848	-15.7%	-14.1%	-26.0%
chain-walk	43 187	32 389	32 209	21 366	-25.0%	-25.4%	-50.5%
pipeline	777 831	733 339	617 842	592 627	-5.7%	-20.6%	-23.8%
record-write	316 080	327 120	328 146	322 073	+3.5%	+3.8%	+1.9%
nested-write	206 807	199 763	194 337	192 229	-3.4%	-6.0%	-7.0%
accumulate	1 431	1 302	1 475	693	-9.0%	+3.1%	-51.6%
interpolation	189 125	189 870	185 263	185 881	+0.4%	-2.0%	-1.7%
map-count	169 777	153 854	154 009	154 667	-9.4%	-9.3%	-8.9%
map-write	304 235	287 730	287 610	282 727	-5.4%	-5.5%	-7.1%
cons-stack	1 957	2 358	1 469	1 168	+20.5%	-24.9%	-40.3%
array-stack	55 770	53 211	54 970	54 073	-4.6%	-1.4%	-3.0%
bankers-queue	84 892	72 820	74 652	64 289	-14.2%	-12.1%	-24.3%
ring-list	3 965	4 260	3 882	3 630	+7.4%	-2.1%	-8.4%
array-front	30 705	31 141	31 359	30 312	+1.4%	+2.1%	-1.3%
buffer-write	160 602	159 767	161 876	158 168	-0.5%	+0.8%	-1.5%
array-write	161 183	159 310	161 219	157 500	-1.2%	+0.0%	-2.3%
buffer-swap-remove	21 758	20 239	19 589	19 337	-7.0%	-10.0%	-11.1%
array-swap-remove	20 174	20 956	19 807	18 765	+3.9%	-1.8%	-7.0%
heap	185 294	172 920	165 477	150 104	-6.7%	-10.7%	-19.0%
heap-sorted	117 014	116 616	117 434	120 055	-0.3%	+0.4%	+2.6%
binary-trees (game)	1 145 914	1 112 809	1 118 069	1 050 380	-2.9%	-2.4%	-8.3%

Allocations: identical in all four variants for every row.

Decision (rule from #59)

At least 5% faster on two named rows, no named row more than 3% slower, conformance green on both back ends.

Variant	Named rows 5% faster	Slowest named row	Verdict
B	list-iterate, pipeline, map-count	record-write +3.5%	no
C	list-iterate, pipeline, map-count	record-write +3.8%	no
D	list-iterate -26.0%, pipeline -23.8%, map-count -8.9%, binary-trees -8.3%	list-index +2.4%	yes

D lands. Conformance on both back ends and the runtime tests: this PR's CI.

Code size

-O2 object of the emitted program.c: list-iterate 1378 bytes text in all variants (+0), pipeline 3937 to 4677 (+740), record-write, list-index, map-count unchanged. Text of runtime/*.c objects summed: 440971 (A) to 443539 (D), +2568 (+0.6%); machine.c +128, task.c +1544.

Test

Locally: no C compiler on this host, so nothing built here; the gates run in this PR's CI. -flto was not tried (out of scope per the issue). The VM's kernel (machine.c) is unchanged and calls the same names, which are now the inline functions over the same slow entry points.

Seed impact: none. No syntax, native, or name looked up by string.

Closes #59 ## What Round P12 (docs/PERFORMANCE.md finding 13 again): the common case of `torb_retain`, `torb_release` and `torb_make_unique` is `static inline` in `runtime/include/torb.h`; every other case is the out-of-line `torb_retain_slow`, `torb_release_slow` and `torb_make_unique_slow` in `runtime/memory.c` (NULL, immortal, shared/threaded, the last count with the drop depth bound, the zero-count panic, the copy). Semantics of `memory.c` are unchanged. Three commits: `fix(benchmarks)` (run.sh --allocations did not link `runtime/os/` and the threads, so the allocation column was empty), `perf(runtime)`, `docs(performance)`. The fix is a prerequisite for the allocation check of this round; it could not be folded into either of the two. ## Measurement One forge CI job, one runner (Linux 6.8, AMD EPYC Genoa, 16 processors, gcc 13.3.0, `-O2`), `benchmarks/run.sh --allocations`, RUNS=9, passes A B C D then D C B A, fastest of the eighteen runs. Variants: A as is, B retain inline, C B plus the unique test, D all three. Base: `main` at ff037c08. The temporary measurement job and script are not part of this branch (they were removed from the history; the job was a push-triggered workflow on this branch's earlier commits). | Program | A us | B us | C us | D us | B | C | D | |---------|-----:|-----:|-----:|-----:|--:|--:|--:| | `arithmetic` | 236 925 | 237 606 | 241 680 | 238 782 | +0.3% | +2.0% | +0.8% | | `call-depth` | 476 931 | 481 882 | 485 981 | 480 944 | +1.0% | +1.9% | +0.8% | | `wrapper` | 113 819 | 114 163 | 113 909 | 113 773 | +0.3% | +0.1% | -0.0% | | `closure` | 198 050 | 199 480 | 198 771 | 197 821 | +0.7% | +0.4% | -0.1% | | `list-index` | 190 037 | 192 543 | 193 971 | 194 653 | +1.3% | +2.1% | +2.4% | | `list-iterate` | 468 418 | 394 924 | 402 555 | 346 848 | -15.7% | -14.1% | **-26.0%** | | `chain-walk` | 43 187 | 32 389 | 32 209 | 21 366 | -25.0% | -25.4% | -50.5% | | `pipeline` | 777 831 | 733 339 | 617 842 | 592 627 | -5.7% | -20.6% | **-23.8%** | | `record-write` | 316 080 | 327 120 | 328 146 | 322 073 | +3.5% | +3.8% | +1.9% | | `nested-write` | 206 807 | 199 763 | 194 337 | 192 229 | -3.4% | -6.0% | -7.0% | | `accumulate` | 1 431 | 1 302 | 1 475 | 693 | -9.0% | +3.1% | -51.6% | | `interpolation` | 189 125 | 189 870 | 185 263 | 185 881 | +0.4% | -2.0% | -1.7% | | `map-count` | 169 777 | 153 854 | 154 009 | 154 667 | -9.4% | -9.3% | **-8.9%** | | `map-write` | 304 235 | 287 730 | 287 610 | 282 727 | -5.4% | -5.5% | -7.1% | | `cons-stack` | 1 957 | 2 358 | 1 469 | 1 168 | +20.5% | -24.9% | -40.3% | | `array-stack` | 55 770 | 53 211 | 54 970 | 54 073 | -4.6% | -1.4% | -3.0% | | `bankers-queue` | 84 892 | 72 820 | 74 652 | 64 289 | -14.2% | -12.1% | -24.3% | | `ring-list` | 3 965 | 4 260 | 3 882 | 3 630 | +7.4% | -2.1% | -8.4% | | `array-front` | 30 705 | 31 141 | 31 359 | 30 312 | +1.4% | +2.1% | -1.3% | | `buffer-write` | 160 602 | 159 767 | 161 876 | 158 168 | -0.5% | +0.8% | -1.5% | | `array-write` | 161 183 | 159 310 | 161 219 | 157 500 | -1.2% | +0.0% | -2.3% | | `buffer-swap-remove` | 21 758 | 20 239 | 19 589 | 19 337 | -7.0% | -10.0% | -11.1% | | `array-swap-remove` | 20 174 | 20 956 | 19 807 | 18 765 | +3.9% | -1.8% | -7.0% | | `heap` | 185 294 | 172 920 | 165 477 | 150 104 | -6.7% | -10.7% | -19.0% | | `heap-sorted` | 117 014 | 116 616 | 117 434 | 120 055 | -0.3% | +0.4% | +2.6% | | `binary-trees` (game) | 1 145 914 | 1 112 809 | 1 118 069 | 1 050 380 | -2.9% | -2.4% | **-8.3%** | Allocations: identical in all four variants for every row. ## Decision (rule from #59) At least 5% faster on two named rows, no named row more than 3% slower, conformance green on both back ends. | Variant | Named rows 5% faster | Slowest named row | Verdict | |---|---|---|---| | B | list-iterate, pipeline, map-count | record-write +3.5% | no | | C | list-iterate, pipeline, map-count | record-write +3.8% | no | | D | list-iterate -26.0%, pipeline -23.8%, map-count -8.9%, binary-trees -8.3% | list-index +2.4% | yes | D lands. Conformance on both back ends and the runtime tests: this PR's CI. ## Code size `-O2` object of the emitted `program.c`: list-iterate 1378 bytes text in all variants (+0), pipeline 3937 to 4677 (+740), record-write, list-index, map-count unchanged. Text of `runtime/*.c` objects summed: 440971 (A) to 443539 (D), +2568 (+0.6%); `machine.c` +128, `task.c` +1544. ## Test Locally: no C compiler on this host, so nothing built here; the gates run in this PR's CI. `-flto` was not tried (out of scope per the issue). The VM's kernel (`machine.c`) is unchanged and calls the same names, which are now the inline functions over the same slow entry points. Seed impact: none. No syntax, native, or name looked up by string.

orion added 3 commits 2026-10-23 22:04:37 +00:00

fix(benchmarks): link runtime/os and the threads into the allocation-counting build ... 18dcea8912

run.sh --allocations built the counted copy from runtime/*.c only, so a runtime that reaches into runtime/os/ or
uses the worker pool no longer linked and the allocation column stayed empty. It links what torb build links.

perf(runtime): inline the fast path of torb_retain, torb_release and torb_make_unique ... 6a2d267ff0

The common case is the few instructions of a static inline function in torb.h: a retain of a block that is neither
shared nor immortal, a release that leaves a count above one, the unique test of make_unique. Every other case calls the
out-of-line torb_retain_slow, torb_release_slow and torb_make_unique_slow in memory.c, so the panic, the immortal rule,
the atomic path and the bound on how deep drops nest stay where they were.

Measured against main in one forge CI job (docs/PERFORMANCE.md finding 13): list-iterate -26%, pipeline -24%,
map-count -9%, binary-trees -8%; no named row is more than 3% slower. Allocations do not change in any row.

docs(performance): redo finding 13 on the code P5 to P7 leave behind 1a2987b4c0

orion force-pushed loop/59-inline-refcount-fastpath from 1a2987b4c0 to 7d4fe624a4 2026-10-23 22:05:29 +00:00 Compare

argus approved these changes 2026-10-23 22:35:05 +00:00

argus left a comment

Copy link

Approved.

• Tests/CI: combined status success on this head (bootstrap, tier A/B, every host emits the same C, playground, pdisc, gates). Behavior-preserving refactor; the existing runtime tests and conformance on both back ends cover it. Windows/arm64/macOS jobs are skipped on PRs by the workflow, not by this change.
• Correctness: checked the inline fast paths against the memory.c originals. torb_retain: non-NULL and count < SHARED, plain +1 (the same result as before, immortal and shared go to the slow path). torb_release: only 1 < count < SHARED is decremented inline, so the last count, the zero-count panic, immortal, shared/atomic and the drop-depth bound all stay in torb_release_slow. torb_make_unique: the (count & ~SHARED) == 1 test equals torb_is_unique, an immortal count masks to 0x7FFFFFFF and still copies. The count is read relaxed, as torb_count_of does. No place takes the address of the three symbols; the C emitter and the VM kernel call them by name, so they pick up the inline versions.
• Scope: matches #59 (rule set before the numbers, variant D meets it); the run.sh --allocations link fix is a stated prerequisite for the allocation check.
• Security/data: none touched; no new dependencies, no storage format change.
• Docs: PERFORMANCE.md (F13, P12 row, status), runtime/README.md and the pdisc sidecar updated.

Non-blocking: the PR body says nothing was built locally; fine since CI is authoritative, but a one-line runtime test that calls the _slow entry points directly (zero-count panic, immortal) would pin the contract that they are complete functions.

Approved. - Tests/CI: combined status success on this head (bootstrap, tier A/B, every host emits the same C, playground, pdisc, gates). Behavior-preserving refactor; the existing runtime tests and conformance on both back ends cover it. Windows/arm64/macOS jobs are skipped on PRs by the workflow, not by this change. - Correctness: checked the inline fast paths against the `memory.c` originals. `torb_retain`: non-NULL and count < SHARED, plain +1 (the same result as before, immortal and shared go to the slow path). `torb_release`: only 1 < count < SHARED is decremented inline, so the last count, the zero-count panic, immortal, shared/atomic and the drop-depth bound all stay in `torb_release_slow`. `torb_make_unique`: the `(count & ~SHARED) == 1` test equals `torb_is_unique`, an immortal count masks to 0x7FFFFFFF and still copies. The count is read relaxed, as `torb_count_of` does. No place takes the address of the three symbols; the C emitter and the VM kernel call them by name, so they pick up the inline versions. - Scope: matches #59 (rule set before the numbers, variant D meets it); the `run.sh --allocations` link fix is a stated prerequisite for the allocation check. - Security/data: none touched; no new dependencies, no storage format change. - Docs: PERFORMANCE.md (F13, P12 row, status), runtime/README.md and the pdisc sidecar updated. Non-blocking: the PR body says nothing was built locally; fine since CI is authoritative, but a one-line runtime test that calls the `_slow` entry points directly (zero-count panic, immortal) would pin the contract that they are complete functions.

argus merged commit 7d4fe624a4 into main 2026-10-23 22:35:06 +00:00

argus deleted branch loop/59-inline-refcount-fastpath 2026-10-23 22:35:06 +00:00

orion referenced this pull request 2026-10-23 22:37:49 +00:00

perf: P12 - F13 again on the code P5 to P7 leave behind #59

Sign in to join this conversation.

Reviewers

No reviewers

argus

Labels

Clear labels   

loop

Backlog item of the project loop (Orion), see CONTRIBUTING

  

ready

Refined to the definition of ready

No labels

loop

ready

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

torbscript/language!444

No description provided.