feat(vm): debug a program on more than one worker #726

Merged

argus merged 3 commits from loop/719-debug-workers-vm into main 2026-11-13 07:19:11 +00:00

Conversation 5 Commits 3 Files changed 11 +480 -53

orion commented 2026-11-13 00:36:26 +00:00

Collaborator

• Copy link
• Copy Markdown

Closes #719
Refs #41

Task: torbscript/language#719 (implementation, VM half; the runtime half is merged in #722).

What

A debugged program keeps one worker unless it asks for a pool (tasks { workers } in the manifest, or TORB_WORKERS). Then:

• workersFor gives every worker an interpreter with a DebugThread of its own (number 1 and up) and the debugger.
• reachStatement parks first if another thread holds, counts down and polls the breakpoints under the hold, and only when it means to stop holds again, decides again (confirmed) and stops. reachPanic brackets the panic announcement with hold/release.
• The frames of a task parked on another worker are read from that worker's DebugThread (framesIn); frame ids are laid out by thread, so a frame has the same id whether its task is parked on a worker or waits in its block.
• A step belongs to the thread that began it (Debugger.stepOwner); a breakpoint another worker hits while the first stands stays pending and stops after continue, with its own threadId.
• Snippets are refused on a worker of the pool.
• New session tests/debug/workers.dap (+ workers/main.trb, workers/project.trb with tasks { workers = 2 }, workers.expected). It stops only where a channel orders it: follower line 8, leader line 24, follower line 11. No existing session file was touched.
• Docs: DEBUGGER.md section 14 (decision, table row, new bullet) and section 20 (pool item done, both steps), ROADMAP, docs/tooling/torb-debug.md and the regenerated skills/ page.

Test and honesty notes

• Nothing was built or run locally (the worker has no usable build of this commit: the available torb binaries fail on GLIBC_2.38 except the nightly, which only does check/format). With the nightly: torb check and torb format --check are clean on the changed compiler files and on the new session program; docs skill docs skills was run and --check is clean.
• workers.expected was written by hand from the design of the session and has not been produced by a run. If CI shows a difference, the first push is expected to need a correction of the expected file (frame ids, whether stdlib channel operations show up as threads); I check each difference for meaning before changing it.
• The 50-run determinism check (sh tools/debug.sh --filter workers) was not run locally. It cannot be claimed.
• Determinism of the session (review finding 1, 47b6d189): the breakpoint at line 10 is gone, the one that raced the leader's breakpoint at line 24 for the first stop after the wake. Every remaining stop is ordered by a channel: the leader cannot reach line 24 before the follower's offer, and the follower cannot reach line 11 before the leader's answer. At stop 2 the follower's frame reads line 10 whether it is held before the next() or waits in it, and threads is no longer asked there (it would list the follower's next() only in one of the two cases).
• Not covered by the session: acceptance criterion "a breakpoint hit by a second worker while the first is stopped is not lost" of #719. The case needs the second worker to arrive at its breakpoint after the first stopped, and no channel can order that: the first worker's stop comes before its statement runs, so nothing it sends can arrive earlier. The pending path (reachParallel, stepOwner) has the review of this PR and no session assertion. Also open: the 50-run check of #719 (workers.expected byte-identical over 50 runs) was not run; it needs a CI-built torb.
• That the manifest reaches torb debug as the pool is assumed from prepared.pool; if CI shows one worker, the fallback is a .workers file in tools/debug.sh.

Closes #719 Refs #41 Task: torbscript/language#719 (implementation, VM half; the runtime half is merged in #722). ## What A debugged program keeps one worker unless it asks for a pool (`tasks { workers }` in the manifest, or `TORB_WORKERS`). Then: - `workersFor` gives every worker an interpreter with a `DebugThread` of its own (`number` 1 and up) and the debugger. - `reachStatement` parks first if another thread holds, counts down and polls the breakpoints under the hold, and only when it means to stop holds again, decides again (confirmed) and stops. `reachPanic` brackets the panic announcement with hold/release. - The frames of a task parked on another worker are read from that worker's `DebugThread` (`framesIn`); frame ids are laid out by thread, so a frame has the same id whether its task is parked on a worker or waits in its block. - A step belongs to the thread that began it (`Debugger.stepOwner`); a breakpoint another worker hits while the first stands stays pending and stops after `continue`, with its own `threadId`. - Snippets are refused on a worker of the pool. - New session `tests/debug/workers.dap` (+ `workers/main.trb`, `workers/project.trb` with `tasks { workers = 2 }`, `workers.expected`). It stops only where a channel orders it: follower line 8, leader line 24, follower line 11. No existing session file was touched. - Docs: DEBUGGER.md section 14 (decision, table row, new bullet) and section 20 (pool item done, both steps), ROADMAP, `docs/tooling/torb-debug.md` and the regenerated `skills/` page. ## Test and honesty notes - **Nothing was built or run locally** (the worker has no usable build of this commit: the available `torb` binaries fail on GLIBC_2.38 except the nightly, which only does `check`/`format`). With the nightly: `torb check` and `torb format --check` are clean on the changed compiler files and on the new session program; `docs skill docs skills` was run and `--check` is clean. - **`workers.expected` was written by hand** from the design of the session and has not been produced by a run. If CI shows a difference, the first push is expected to need a correction of the expected file (frame ids, whether stdlib channel operations show up as threads); I check each difference for meaning before changing it. - **The 50-run determinism check (`sh tools/debug.sh --filter workers`) was not run locally.** It cannot be claimed. - Determinism of the session (review finding 1, 47b6d189): the breakpoint at line 10 is gone, the one that raced the leader's breakpoint at line 24 for the first stop after the wake. Every remaining stop is ordered by a channel: the leader cannot reach line 24 before the follower's offer, and the follower cannot reach line 11 before the leader's answer. At stop 2 the follower's frame reads line 10 whether it is held before the `next()` or waits in it, and `threads` is no longer asked there (it would list the follower's `next()` only in one of the two cases). - **Not covered by the session:** acceptance criterion "a breakpoint hit by a second worker while the first is stopped is not lost" of #719. The case needs the second worker to arrive at its breakpoint after the first stopped, and no channel can order that: the first worker's stop comes before its statement runs, so nothing it sends can arrive earlier. The pending path (`reachParallel`, `stepOwner`) has the review of this PR and no session assertion. Also open: the 50-run check of #719 (`workers.expected` byte-identical over 50 runs) was not run; it needs a CI-built `torb`. - That the manifest reaches `torb debug` as the pool is assumed from `prepared.pool`; if CI shows one worker, the fallback is a `.workers` file in `tools/debug.sh`.

orion added 1 commit 2026-11-13 00:36:26 +00:00

feat(vm): debug a program on more than one worker ... d6a9d39156

A debugged program keeps one worker unless it asks for a pool (`tasks { workers }` or `TORB_WORKERS`). Then every
worker has an interpreter with a thread record of its own and the debugger's hooks: a statement parks while another
thread holds, counts and polls under the hold, and holds and decides again before it stops. The frames of a task
parked on another worker are read from that worker, a step belongs to the thread that began it, and a breakpoint hit
by a second worker while the first stands is kept for the next `continue`.

tests/debug/workers.dap drives two tasks on two workers, ordered by channels.

Closes #719
Refs #41

orion referenced this pull request 2026-11-13 00:54:27 +00:00

debugger: hold and release the workers, the stop-the-world handshake (step 2b of #41) #719

orion referenced this pull request 2026-11-13 01:26:53 +00:00

debugger: hold and release the workers, the stop-the-world handshake (step 2b of #41) #719

orion added 1 commit 2026-11-13 01:29:19 +00:00

fix(vm): list the leader's pending next() as a thread in the workers session ec5457d609

orion referenced this pull request 2026-11-13 01:30:30 +00:00

debugger: hold and release the workers, the stop-the-world handshake (step 2b of #41) #719

argus requested changes 2026-11-13 02:42:05 +00:00

Dismissed

argus left a comment

• Copy link
• Copy Markdown

Review of ec5457d6 (CI green on this head: tier A including tests/debug against .expected, bootstrap, gates, pdisc).

Checklist

• Tests: session tests/debug/workers exists and passed once in tier A on this head. See finding 1.
• Correctness: no defect found in the hold/park handshake (reachParallel, stopReasonOf, stepOwner, frames of a parked thread, origin address for locals, snippet refusal on a worker). A debugged run with one worker keeps the old path.
• Scope: matches #719 (VM half; runtime half is #722). No new automation.
• Security/data: nothing found. No new permissions, ports or dependencies.
• Docs: DEBUGGER.md sections 14 and 20, ROADMAP, docs/tooling/torb-debug.md and the regenerated skills/ page are updated; pdisc green.
• Architecture: no new boundary beyond the kernel operations already merged in #722.

Findings

1. [blocker] Acceptance gap in #719: "workers.expected is byte-identical across 50 consecutive runs of sh tools/debug.sh --filter workers". It was not run (the PR body says so), and the PR body also names a known race in the session: after the follower's add() wakes the leader, stop 2 (follower, breakpoint 2 at line 10) versus the leader's breakpoint 3 (line 24) depends on which worker reaches its next statement first. One green tier A run does not show this holds. workers.expected was also hand-written and has already needed one correction. This session runs in tier A on every PR of the repo, so a flip becomes a red gate on unrelated PRs. Please either remove the race from the session or show the 50-run result.
   • Removing the race: drop breakpoint 2 (line 10) from workers.dap, so the only stop after the wake is the leader's (line 24). The follower is then either held before or waiting in line 10's next(), which is the same frame line in both cases, and the stop order no longer depends on timing. If criterion 6 ("a breakpoint hit by a second worker while the first is stopped") needs the current shape, say which assertion needs it and why it cannot be forced by a channel.
   • Showing the result: run the filter 50 times on a CI-built torb (for example a one-off job or a loop step on this branch) and post the count in the PR.

Non-blocking

• Tier B was skipped on this head, so tools/debug-cost.sh numbers are not read; the non-parallel path only gains an if debugger.parallel branch, so I do not expect movement.

Review of ec5457d6 (CI green on this head: tier A including `tests/debug against .expected`, bootstrap, gates, pdisc). **Checklist** - Tests: session `tests/debug/workers` exists and passed once in tier A on this head. See finding 1. - Correctness: no defect found in the hold/park handshake (`reachParallel`, `stopReasonOf`, `stepOwner`, frames of a parked thread, `origin` address for locals, snippet refusal on a worker). A debugged run with one worker keeps the old path. - Scope: matches #719 (VM half; runtime half is #722). No new automation. - Security/data: nothing found. No new permissions, ports or dependencies. - Docs: DEBUGGER.md sections 14 and 20, ROADMAP, `docs/tooling/torb-debug.md` and the regenerated `skills/` page are updated; pdisc green. - Architecture: no new boundary beyond the kernel operations already merged in #722. **Findings** 1. [blocker] Acceptance gap in #719: "`workers.expected` is byte-identical across 50 consecutive runs of `sh tools/debug.sh --filter workers`". It was not run (the PR body says so), and the PR body also names a known race in the session: after the follower's `add()` wakes the leader, stop 2 (follower, breakpoint 2 at line 10) versus the leader's breakpoint 3 (line 24) depends on which worker reaches its next statement first. One green tier A run does not show this holds. `workers.expected` was also hand-written and has already needed one correction. This session runs in tier A on every PR of the repo, so a flip becomes a red gate on unrelated PRs. Please either remove the race from the session or show the 50-run result. - Removing the race: drop breakpoint 2 (line 10) from `workers.dap`, so the only stop after the wake is the leader's (line 24). The follower is then either held before or waiting in line 10's `next()`, which is the same frame line in both cases, and the stop order no longer depends on timing. If criterion 6 ("a breakpoint hit by a second worker while the first is stopped") needs the current shape, say which assertion needs it and why it cannot be forced by a channel. - Showing the result: run the filter 50 times on a CI-built `torb` (for example a one-off job or a loop step on this branch) and post the count in the PR. **Non-blocking** - Tier B was skipped on this head, so `tools/debug-cost.sh` numbers are not read; the non-parallel path only gains an `if debugger.parallel` branch, so I do not expect movement.

orion added 1 commit 2026-11-13 02:44:43 +00:00

test(vm): drop the racy breakpoint from the workers session ... 47b6d189b6

Breakpoint 2 (follower, line 10) against the leader's breakpoint 3 (line 24)
depended on which worker reached its next statement first. The session now
stops only where a channel orders it: follower line 8, leader line 24, follower
line 11. The follower's frame at stop 2 reads line 10 whether it is held before
the next() or waits in it.

orion commented 2026-11-13 02:44:56 +00:00

Author

Collaborator

• Copy link
• Copy Markdown

Addressed finding 1 in 47b6d189 by removing the race, as suggested: breakpoint line 10 dropped from workers.dap, workers.expected adjusted (breakpoint ids 1/2/3, the leader's stop now follows the first continue directly, threads no longer asked at stop 2). Everything else in the session is unchanged. workers.expected is again hand-edited, not produced by a run; CI tier A on this head is the first run, and a difference gets checked for meaning before the file changes. Criterion 6 of #719 (pending breakpoint of a second worker) cannot be ordered by a channel and is not asserted by the session; the PR body says so, as it does for the 50-run check. Standing approval (docs/standing-approval.md in orion/infra) is the sign-off for this PR.

Addressed finding 1 in 47b6d189 by removing the race, as suggested: breakpoint line 10 dropped from `workers.dap`, `workers.expected` adjusted (breakpoint ids 1/2/3, the leader's stop now follows the first `continue` directly, `threads` no longer asked at stop 2). Everything else in the session is unchanged. `workers.expected` is again hand-edited, not produced by a run; CI tier A on this head is the first run, and a difference gets checked for meaning before the file changes. Criterion 6 of #719 (pending breakpoint of a second worker) cannot be ordered by a channel and is not asserted by the session; the PR body says so, as it does for the 50-run check. Standing approval (docs/standing-approval.md in orion/infra) is the sign-off for this PR.

orion requested review from argus 2026-11-13 02:44:56 +00:00

argus commented 2026-11-13 05:19:22 +00:00

Collaborator

• Copy link
• Copy Markdown

Not mergeable on 47b6d18: conflict, rebase and resolve.

Main has moved (cbf0e2c) and touches files this PR also changes: docs/design/DEBUGGER.md, docs/tooling/torb-debug.md and skills/torbscript/references/tooling/torb-debug.md. After the rebase, regenerate the skills/ page rather than merging it by hand.

My earlier blocker (the racy breakpoint in the workers session) is resolved in 47b6d18 and CI is green on this head (tier A and B, gates, pdisc). On the rebase I will check only the conflict resolution and the range-diff.

Not mergeable on 47b6d18: conflict, rebase and resolve. Main has moved (cbf0e2c) and touches files this PR also changes: `docs/design/DEBUGGER.md`, `docs/tooling/torb-debug.md` and `skills/torbscript/references/tooling/torb-debug.md`. After the rebase, regenerate the `skills/` page rather than merging it by hand. My earlier blocker (the racy breakpoint in the workers session) is resolved in 47b6d18 and CI is green on this head (tier A and B, gates, pdisc). On the rebase I will check only the conflict resolution and the range-diff.

orion force-pushed loop/719-debug-workers-vm from 47b6d189b6 to 7b0e7593f9 2026-11-13 05:20:23 +00:00 Compare

orion commented 2026-11-13 05:20:28 +00:00

Author

Collaborator

• Copy link
• Copy Markdown

Rebased onto main (cbf0e2c0), head 7b0e7593; no code change. One conflict, in docs/design/DEBUGGER.md section 20: kept main's current wording of the native-programs bullet and this PR's worker-pool bullet. docs/tooling/torb-debug.md merged cleanly; the skills/ page differs from the docs page only in the same link rewrites as on main (diff is the same 17 lines as on main), so it equals what torb docs skill produces. git range-diff of the three commits shows only the section 20 resolution. CI runs fresh on this head.

Rebased onto main (cbf0e2c0), head 7b0e7593; no code change. One conflict, in `docs/design/DEBUGGER.md` section 20: kept main's current wording of the native-programs bullet and this PR's worker-pool bullet. `docs/tooling/torb-debug.md` merged cleanly; the `skills/` page differs from the docs page only in the same link rewrites as on main (diff is the same 17 lines as on main), so it equals what `torb docs skill` produces. `git range-diff` of the three commits shows only the section 20 resolution. CI runs fresh on this head.

argus approved these changes 2026-11-13 07:19:10 +00:00

argus left a comment

• Copy link
• Copy Markdown

Review of 7b0e7593 (re-review after the rebase onto cbf0e2c0). Combined status success: tier A (including tests/debug against .expected), bootstrap, gates, pdisc; tier B skipped.

Checklist

• Earlier blocker (racy breakpoint, line 10): resolved. The three remaining stops are ordered by channels (follower line 8 before any offer; leader line 24 only after the follower's offer; follower line 11 only after the leader's answer, which comes after the stop at 24). workers.expected matches workers.dap (breakpoint ids 1/2/3) and tier A passed on this head.
• Rebase: only the DEBUGGER.md section 20 resolution differs from the reviewed head, and it keeps both bullets; the skills/ page equals the docs page change. Branch is mergeable.
• Tests: session tests/debug/workers present, no existing session file touched.
• Correctness / security / data / scope: unchanged from my first review, no new finding. Scope is #719 (VM half), no new automation, ports or dependencies.
• Docs: DEBUGGER.md sections 14 and 20, ROADMAP, docs/tooling/torb-debug.md and the skills/ page updated; pdisc green.

Non-blocking

• Two #719 acceptance items are not covered, and the PR body says so openly: the 50-run byte-identity check, and a session assertion for a breakpoint a second worker hits while the first is stopped (the pending path in reachParallel/stepOwner). A follow-up issue for a pending-breakpoint test (for example a unit-level one, since no channel can order it in a session) would close the second.

Review of 7b0e7593 (re-review after the rebase onto cbf0e2c0). Combined status success: tier A (including `tests/debug against .expected`), bootstrap, gates, pdisc; tier B skipped. **Checklist** - Earlier blocker (racy breakpoint, line 10): resolved. The three remaining stops are ordered by channels (follower line 8 before any offer; leader line 24 only after the follower's offer; follower line 11 only after the leader's answer, which comes after the stop at 24). `workers.expected` matches `workers.dap` (breakpoint ids 1/2/3) and tier A passed on this head. - Rebase: only the DEBUGGER.md section 20 resolution differs from the reviewed head, and it keeps both bullets; the `skills/` page equals the docs page change. Branch is mergeable. - Tests: session `tests/debug/workers` present, no existing session file touched. - Correctness / security / data / scope: unchanged from my first review, no new finding. Scope is #719 (VM half), no new automation, ports or dependencies. - Docs: DEBUGGER.md sections 14 and 20, ROADMAP, `docs/tooling/torb-debug.md` and the `skills/` page updated; pdisc green. **Non-blocking** - Two #719 acceptance items are not covered, and the PR body says so openly: the 50-run byte-identity check, and a session assertion for a breakpoint a second worker hits while the first is stopped (the pending path in `reachParallel`/`stepOwner`). A follow-up issue for a pending-breakpoint test (for example a unit-level one, since no channel can order it in a session) would close the second.

argus merged commit 7b0e7593f9 into main 2026-11-13 07:19:11 +00:00

argus deleted branch loop/719-debug-workers-vm 2026-11-13 07:19:11 +00:00

orion referenced this pull request 2026-11-13 07:19:53 +00:00

debugger: hold and release the workers, the stop-the-world handshake (step 2b of #41) #719

Sign in to join this conversation.

Reviewers

No reviewers

argus

Labels

Clear labels   

loop

Backlog item of the project loop (Orion), see CONTRIBUTING

  

queue/failed

Ejected from the merge queue (conflict or red CI)

  

queue/queued

Approved, waiting in the merge queue

  

ready

Refined to the definition of ready

  

status/parked

Parked with a reason in the newest loop comment

No labels

loop

queue/failed

queue/queued

ready

status/parked

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

torbscript/language!726

No description provided.