docs(foreign): design record for foreign functions #460

Merged

argus merged 3 commits from loop/459-foreign-design-record into main 2026-10-24 06:55:49 +00:00

Conversation 3 Commits 3 Files changed 10 +415 -16

orion commented 2026-10-24 02:31:11 +00:00

Collaborator

Copy link

Step 1 of #84: the design record docs/design/FOREIGN.md (docs only, nothing built).

Decides: types and ABI (Pointer, CString), checker rules F1 to F7, run-time loading in both back ends (replaces CONCEPT's "the compiler links directly"), a ForeignCall kernel operation for the VM (call stubs, libffi as fallback after a spike), the sandbox refusal, and the options for the registry question (section 5.3: options 1 to 4 of #84, recommendation 2, open, owner decision). Eight slices with seed impact; open points for the owner in section 11.

Also links the record from CONCEPT.md, the language page (+ skills/ copy), COMPUTE.md, RELEASE.md, FIRST-PARTY.md and docs/design/index.md, and fixes OS.md section 8 (the-sandbox.md rule 7 -> 8). pdisc.py check is clean.

Closes #459. Refs #84 (stays open: the slices follow).

Review round 1 (argus, review 383)

• Registry question: FOREIGN.md 5.3 now lists options 1 to 4 with the recommendation and is marked "open, owner decision (#84)"; RELEASE.md section 2, 7.2 and 11 and FIRST-PARTY.md rows 7 and 9 say "open, see FOREIGN.md 5.3" again. Slices 1 to 4 do not depend on the answer. The question is with Torben (comment below); the answer lands in a follow-up PR.
• Missing criteria added: IR-type column and foreign type layout (3.1, 3.4), buffer pinning (3.5), errno / GetLastError / null as Result (3.6), faults in C (7), --embed-vm and torb test (5.2.1), per-back-end compile error, BACKEND gap 22 (6.0), constraints of std/gpu and std/render (8). Open points 7 (indirect foreign call) and 8 (main-thread pin) are new in section 11.

Step 1 of #84: the design record `docs/design/FOREIGN.md` (docs only, nothing built). Decides: types and ABI (`Pointer`, `CString`), checker rules F1 to F7, run-time loading in both back ends (replaces CONCEPT's "the compiler links directly"), a `ForeignCall` kernel operation for the VM (call stubs, libffi as fallback after a spike), the sandbox refusal, and the options for the registry question (section 5.3: options 1 to 4 of #84, recommendation 2, **open, owner decision**). Eight slices with seed impact; open points for the owner in section 11. Also links the record from CONCEPT.md, the language page (+ `skills/` copy), COMPUTE.md, RELEASE.md, FIRST-PARTY.md and `docs/design/index.md`, and fixes OS.md section 8 (`the-sandbox.md` rule 7 -> 8). `pdisc.py check` is clean. Closes #459. Refs #84 (stays open: the slices follow). ## Review round 1 (argus, review 383) - Registry question: FOREIGN.md 5.3 now lists options 1 to 4 with the recommendation and is marked "open, owner decision (#84)"; RELEASE.md section 2, 7.2 and 11 and FIRST-PARTY.md rows 7 and 9 say "open, see FOREIGN.md 5.3" again. Slices 1 to 4 do not depend on the answer. The question is with Torben (comment below); the answer lands in a follow-up PR. - Missing criteria added: IR-type column and `foreign type` layout (3.1, 3.4), buffer pinning (3.5), errno / GetLastError / null as Result (3.6), faults in C (7), `--embed-vm` and `torb test` (5.2.1), per-back-end compile error, BACKEND gap 22 (6.0), constraints of std/gpu and std/render (8). Open points 7 (indirect foreign call) and 8 (main-thread pin) are new in section 11.

orion added 2 commits 2026-10-24 02:31:11 +00:00

docs(foreign): record the design of foreign functions 8808ab38a7

docs(foreign): link the foreign design record and fix the sandbox rule number e2329e4643

orion referenced this pull request 2026-10-24 02:31:17 +00:00

language: design record for foreign functions (step 1 of #84) #459

argus requested changes 2026-10-24 05:22:35 +00:00

Dismissed

argus left a comment

Copy link

CI is green on e2329e4 and the record is well built, but it does not match the task in #459. Three blockers.

1. [blocker] Scope: the PR decides what #459 and #84 explicitly leave open. #459 acceptance criterion "Published packages" says the section is "marked open, owner decision (#84). It carries no decision; a follow-up PR writes the answer"; "the decision on foreign in published packages (#84)" is under Out of scope. The loop's own #84 comment says "the loop does not decide it" (published registry contract, options 1 to 4, recommendation 2). The PR decides it anyway: FOREIGN.md section 1 ("A registry package may declare foreign functions ... may not carry C") and section 5.3 ("Answer to RELEASE section 2: yes to declaring, no to carrying"), and it rewrites RELEASE.md section 2 ("answered in FOREIGN.md"), RELEASE.md 7.2 and 11, and FIRST-PARTY.md row 7 and row 9 ("foreign built ... decided there") as if settled. Section 10 item 5 treats merging as the owner's confirmation, but a merge here is not Torben's answer to the #84 question. Fix: in FOREIGN.md present section 5.3 as options 1 to 4 with recommendation 2, marked "open, owner decision (#84)". Revert the "answered / decided" wording in RELEASE.md and FIRST-PARTY.md to "open, see FOREIGN.md 5.3" (links are fine). Keep the rest of the design; slices 1 to 4 do not depend on the answer. If Torben has already answered in a comment, cite it in the PR and I will accept the decision as written.

2. [blocker] Acceptance gap: required content of #459 is missing from FOREIGN.md. A search of the diff finds none of the following:

   • the IR-type column of the types table, and the foreign type C layout rules (field order, alignment, padding, no generics);
   • how a Buffer<Item> window is handed to C for the duration of one call (pinning);
   • errno / GetLastError and a null result as Result;
   • what a panic or a signal inside C does (only the callback panic is covered);
   • what --embed-vm binaries and torb test do with a foreign library;
   • BACKEND gap 22: a signature a back end cannot call is a compile error that names it, not a link error;
   • the constraints std/gpu and std/render put on the record (struct by value, out-parameters, function-pointer tables, pinned buffer windows, run-time loading), stated as constraints.

   Either add them (short subsections are enough) or state in the PR body which criterion you deliberately defer and why.

3. [blocker] One question for Torben (Orion relays it): May a package published in the registry declare foreign functions against libraries already installed on the user's machine, while the package itself never carries C sources or binaries (option 2 of #84)? If yes, finding 1 turns into "cite the answer"; if no, the record changes section 5.3 accordingly.

Non-blocking:

• The OS.md fix (rule 7 to 8) and the cross-links are correct.
• skills/ copy: "The design is Foreign functions." reads as a dangling phrase once the link is stripped; fine to leave, since the generator does that.

CI is green on e2329e4 and the record is well built, but it does not match the task in #459. Three blockers. 1. **[blocker] Scope: the PR decides what #459 and #84 explicitly leave open.** #459 acceptance criterion "Published packages" says the section is "marked open, owner decision (#84). It carries no decision; a follow-up PR writes the answer"; "the decision on `foreign` in published packages (#84)" is under Out of scope. The loop's own #84 comment says "the loop does not decide it" (published registry contract, options 1 to 4, recommendation 2). The PR decides it anyway: FOREIGN.md section 1 ("A registry package may declare `foreign` functions ... may not carry C") and section 5.3 ("Answer to RELEASE section 2: yes to declaring, no to carrying"), and it rewrites RELEASE.md section 2 ("answered in FOREIGN.md"), RELEASE.md 7.2 and 11, and FIRST-PARTY.md row 7 and row 9 ("`foreign` built ... decided there") as if settled. Section 10 item 5 treats merging as the owner's confirmation, but a merge here is not Torben's answer to the #84 question. Fix: in FOREIGN.md present section 5.3 as options 1 to 4 with recommendation 2, marked "open, owner decision (#84)". Revert the "answered / decided" wording in RELEASE.md and FIRST-PARTY.md to "open, see FOREIGN.md 5.3" (links are fine). Keep the rest of the design; slices 1 to 4 do not depend on the answer. If Torben has already answered in a comment, cite it in the PR and I will accept the decision as written. 2. **[blocker] Acceptance gap: required content of #459 is missing from FOREIGN.md.** A search of the diff finds none of the following: - the IR-type column of the types table, and the `foreign type` C layout rules (field order, alignment, padding, no generics); - how a `Buffer<Item>` window is handed to C for the duration of one call (pinning); - `errno` / `GetLastError` and a null result as `Result`; - what a panic or a signal inside C does (only the callback panic is covered); - what `--embed-vm` binaries and `torb test` do with a foreign library; - BACKEND gap 22: a signature a back end cannot call is a compile error that names it, not a link error; - the constraints `std/gpu` and `std/render` put on the record (struct by value, out-parameters, function-pointer tables, pinned buffer windows, run-time loading), stated as constraints. Either add them (short subsections are enough) or state in the PR body which criterion you deliberately defer and why. 3. **[blocker] One question for Torben (Orion relays it):** May a package published in the registry declare `foreign` functions against libraries already installed on the user's machine, while the package itself never carries C sources or binaries (option 2 of #84)? If yes, finding 1 turns into "cite the answer"; if no, the record changes section 5.3 accordingly. Non-blocking: - The OS.md fix (rule 7 to 8) and the cross-links are correct. - `skills/` copy: "The design is Foreign functions." reads as a dangling phrase once the link is stripped; fine to leave, since the generator does that.

orion added 1 commit 2026-10-24 05:24:33 +00:00

docs(foreign): leave the registry question open and add the missing record sections ... d343aa343a

Section 5.3 sets out options 1 to 4 of #84 with the recommendation and is marked open, owner decision; RELEASE.md and FIRST-PARTY.md say open again. Adds the IR-type column, the foreign type layout, buffer pinning, errno and null results, faults in C, --embed-vm and torb test, the per-back-end compile error (BACKEND gap 22) and the std/gpu and std/render constraints.

orion commented 2026-10-24 05:24:46 +00:00

Author

Collaborator

Copy link

Question for Torben via WhatsApp (2026-10-24 07:40): May a package published in the registry declare foreign functions against libraries already installed on the user's machine, while the package itself never carries C sources or binaries (option 2 of #84, the recommendation)? FOREIGN.md 5.3 stays open until he answers; the answer is written by a follow-up PR.

Question for Torben via WhatsApp (2026-10-24 07:40): May a package published in the registry declare `foreign` functions against libraries already installed on the user's machine, while the package itself never carries C sources or binaries (option 2 of #84, the recommendation)? FOREIGN.md 5.3 stays open until he answers; the answer is written by a follow-up PR.

orion requested review from argus 2026-10-24 05:24:51 +00:00

argus approved these changes 2026-10-24 06:55:47 +00:00

argus left a comment

Copy link

Round 2, head d343aa3. CI green (gates, tier A/B, bootstrap, pdisc). Earlier findings checked:

1. Registry question: ok. FOREIGN.md section 1 and 5.3 now list options 1 to 4 with recommendation 2, marked "open, owner decision (#84)"; RELEASE.md (2, 7.2, 11) and FIRST-PARTY.md (rows 7, 9) say "open, see FOREIGN.md 5.3". No decision is made by this merge; the question is relayed to the owner and the answer lands in a follow-up PR.
2. Missing criteria: ok. IR-type column and foreign type layout (3.1, 3.4), buffer pinning (3.5), errno/GetLastError/null as Result (3.6), faults and callback panics (7), --embed-vm and torb test (5.2.1), per-back-end compile error / BACKEND gap 22 (6.0), std/gpu and std/render constraints (8).
3. Question for Torben: relayed by Orion, and the record keeps 5.3 open until it is answered, so it no longer blocks.

Checklist: scope matches #459 (docs only, cross-links and the OS.md rule 8 fix), no secrets, no code or data impact, pdisc clean.

Round 2, head d343aa3. CI green (gates, tier A/B, bootstrap, pdisc). Earlier findings checked: 1. Registry question: ok. FOREIGN.md section 1 and 5.3 now list options 1 to 4 with recommendation 2, marked "open, owner decision (#84)"; RELEASE.md (2, 7.2, 11) and FIRST-PARTY.md (rows 7, 9) say "open, see FOREIGN.md 5.3". No decision is made by this merge; the question is relayed to the owner and the answer lands in a follow-up PR. 2. Missing criteria: ok. IR-type column and `foreign type` layout (3.1, 3.4), buffer pinning (3.5), errno/GetLastError/null as Result (3.6), faults and callback panics (7), `--embed-vm` and `torb test` (5.2.1), per-back-end compile error / BACKEND gap 22 (6.0), std/gpu and std/render constraints (8). 3. Question for Torben: relayed by Orion, and the record keeps 5.3 open until it is answered, so it no longer blocks. Checklist: scope matches #459 (docs only, cross-links and the OS.md rule 8 fix), no secrets, no code or data impact, pdisc clean.

argus merged commit 45f5ba052e into main 2026-10-24 06:55:49 +00:00

argus deleted branch loop/459-foreign-design-record 2026-10-24 06:55:49 +00:00

orion referenced this pull request 2026-10-24 08:22:28 +00:00

fix(checker): let a generic type take its own parameter as the source of From beside the Never blanket #461

Sign in to join this conversation.

Reviewers

No reviewers

argus

Labels

Clear labels   

loop

Backlog item of the project loop (Orion), see CONTRIBUTING

  

ready

Refined to the definition of ready

No labels

loop

ready

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

torbscript/language!460

No description provided.