perf: parse the files of a program in parallel

orion commented

2026-10-08 14:25:24 +00:00

Collaborator

Goal

The parse is two thirds of a small build. The files of a program are independent: parse them in parallel with std/parallel and hand the trees back in input order.

Design record

PERFORMANCE.md 9.6 ("The parse, still"). Part of #144. Keeping the parsed std between runs is a different lever with public behaviour (cache location, invalidation); it is not part of this issue.

Acceptance criteria

torb parse, check and build produce the same tokens, trees (torb tokens, torb ast over every .trb file of the repository) and C (tools/agree.sh emit hashes) as main; the order of diagnostics is unchanged.
torb parse std compiler tests (830 files) is faster than on main in the alternating-run comparison on the CI runner, and the small-build case torb check tests/conformance/adts.trb is not slower. If the tree crossing back as a copy eats the gain, the PR is not opened and the numbers go into PERFORMANCE.md 9.7 as "measured and not taken".
A test with a failing file among several shows the same error as sequential parsing.
sh tools/gates.sh a is green in forge CI.

Measurement

Measured in CI, no local bootstrap: the forge CI job for the PR runs sh tools/gates.sh a; the before/after processor times come from torb check tests/conformance/adts.trb, torb build --emit-c tests/conformance/adts.trb and torb build --emit-c ./compiler run with the main binary and the PR binary alternating (3 runs, mean, as in PERFORMANCE.md 9.5), printed in the CI log and copied into PERFORMANCE.md 9.7. CI machines are noisy, so the ratio of alternating runs counts, not absolute seconds. Note: the CI runner's core count bounds the gain; record the core count in the log.

Seed impact

None expected (std/parallel exists). The compiler itself is then built with the seed, so it may only use what the current seed's std/parallel offers; if it does not, split off the seed step. One PR.

## Goal The parse is two thirds of a small build. The files of a program are independent: parse them in parallel with `std/parallel` and hand the trees back in input order. ## Design record [PERFORMANCE.md 9.6](https://git.torb.dev/torbscript/language/src/branch/main/docs/PERFORMANCE.md) ("The parse, still"). Part of #144. Keeping the parsed `std` between runs is a different lever with public behaviour (cache location, invalidation); it is not part of this issue. ## Acceptance criteria - [ ] `torb parse`, `check` and `build` produce the same tokens, trees (`torb tokens`, `torb ast` over every `.trb` file of the repository) and C (`tools/agree.sh emit` hashes) as `main`; the order of diagnostics is unchanged. - [ ] `torb parse std compiler tests` (830 files) is faster than on `main` in the alternating-run comparison on the CI runner, and the small-build case `torb check tests/conformance/adts.trb` is not slower. If the tree crossing back as a copy eats the gain, the PR is not opened and the numbers go into PERFORMANCE.md 9.7 as "measured and not taken". - [ ] A test with a failing file among several shows the same error as sequential parsing. - [ ] `sh tools/gates.sh a` is green in forge CI. ## Measurement Measured in CI, no local bootstrap: the forge CI job for the PR runs `sh tools/gates.sh a`; the before/after processor times come from `torb check tests/conformance/adts.trb`, `torb build --emit-c tests/conformance/adts.trb` and `torb build --emit-c ./compiler` run with the `main` binary and the PR binary alternating (3 runs, mean, as in PERFORMANCE.md 9.5), printed in the CI log and copied into PERFORMANCE.md 9.7. CI machines are noisy, so the ratio of alternating runs counts, not absolute seconds. Note: the CI runner's core count bounds the gain; record the core count in the log. ## Seed impact None expected (`std/parallel` exists). The compiler itself is then built with the seed, so it may only use what the current seed's `std/parallel` offers; if it does not, split off the seed step. One PR.

orion added the

loop

label

2026-10-08 14:25:24 +00:00

orion referenced this issue

2026-10-08 14:25:37 +00:00

perf: the next round of build speed (PERFORMANCE 9.6) #144

orion added the

ready

label

2026-10-08 14:25:37 +00:00

orion referenced this issue

2026-10-08 14:25:37 +00:00

perf: the next round of build speed (PERFORMANCE 9.6) #144

orion commented

2026-10-08 15:19:44 +00:00

Author

Collaborator

[loop:torbscript] claim implement by session:loop-torbscript-20261008 until 2026-10-08T21:19Z

orion commented

2026-10-08 16:35:33 +00:00

Author

Collaborator

[loop:torbscript] release implement: the worker of the previous claim ended without a commit or a branch on the forge, so the claim is void; a new worker takes the issue below

orion commented

2026-10-08 16:35:34 +00:00

Author

Collaborator

[loop:torbscript] claim implement by session:loop-torbscript-20261008 until 2026-10-08T22:35Z

orion referenced this issue from a pull request that will close it,

2026-10-08 16:55:56 +00:00

perf: parse the files of a program in parallel #214

orion commented

2026-10-08 17:14:31 +00:00

Author

Collaborator

[loop:torbscript] implement: PR #214 (branch loop/206-parallel-parse, head 578a581b) is open: files of a program parse in parallel via std/parallel, trees and diagnostics in input order. The worker session was killed before it read the forge CI result; CI on the PR is running. Red CI on the PR starts a fix round. Worker usage: not reported (session killed). next: waiting (forge CI on PR #214)

[loop:torbscript] implement: PR https://git.torb.dev/torbscript/language/pulls/214 (branch `loop/206-parallel-parse`, head 578a581b) is open: files of a program parse in parallel via `std/parallel`, trees and diagnostics in input order. The worker session was killed before it read the forge CI result; CI on the PR is running. Red CI on the PR starts a fix round. Worker usage: not reported (session killed). next: waiting (forge CI on PR #214)

orion commented

2026-10-08 17:52:38 +00:00

Author

Collaborator

[loop:torbscript] claim implement by session:loop-torbscript-20261008 until 2026-10-08T23:52Z