language: design record for foreign functions (step 1 of #84)

orion commented

2026-10-24 02:16:17 +00:00

Collaborator

Context

Step 1 of #84 (language: the design of foreign functions; COMPUTE.md section 2, ROADMAP "Computation and machine learning", CONCEPT "Foreign Functions (Draft)"). foreign is parsed and its block shape is checked (checkForeignShape, compiler/src/semantics/checker/declaration.trb), the package summary already reports it as the capability "foreign functions" (compiler/src/package/archive.trb), and the sandbox never grants it (docs/language/configuration/the-sandbox.md, rule 8). But nothing links or calls a foreign function, Pointer<Value> and CString are declared nowhere in std/, and no design record exists: CONCEPT is a draft with eight bullets, docs/language/extensibility/foreign-functions.md repeats them as "planned", and std/gpu (COMPUTE.md section 2: "then the design of foreign functions; then std/gpu") waits for it. This step is documentation only.

Goal

A design record docs/design/FOREIGN.md that specifies foreign functions end to end (types, ABI mapping, how each back end calls, library resolution, ownership, callbacks, errors) and cuts the build into slices, each one PR. The one question the record cannot answer itself, whether a published package may declare foreign functions and carry C sources, is an owner decision tracked in #84; the record states the options and leaves that section marked open.

Acceptance criteria

docs/design/FOREIGN.md exists, headed and indexed like the other records (status line "designed, not built", an entry in docs/design/index.md); CONCEPT "Foreign Functions (Draft)", docs/language/extensibility/foreign-functions.md, COMPUTE.md (layer 0 and section 2), RELEASE.md section 2 and FIRST-PARTY.md row 7 link it; docs/ regenerates skills/ (torb docs skill docs skills, a tier A gate).
Types and ABI: a table from each foreign type to its C type and to the IR type it lowers to: the sized numbers, Bool, Pointer<Value>, CString, foreign type (C layout: field order, alignment, padding, no generics), and what is refused (String, Int without a size, closures other than the callback form, generics). Says in which std package Pointer and CString are declared, who may construct and dereference a pointer, how a String becomes a CString (copy, NUL check, lifetime of the call) and how a Buffer<Item> window is handed to C for the duration of one call. Each is a numbered decision with its reason.
How the back ends call: the C back end and the VM, both reading the same IR (VM.md section 6, BACKEND). Compares at least: run-time loading (dlopen/LoadLibraryW, the pattern runtime/os/windows.c and runtime/platform.c already use) against link-time linking (CONCEPT's "the compiler links directly"), and, for the VM's "generic trampoline", an in-tree call stub per ABI (SysV x86-64, Win64, AAPCS64) against libffi against generating a C stub per signature. Decides one, with the number of supported tier 1 targets, the dependency and licence cost (RELEASE.md section 7.11: no large C files in every build), and what happens on a signature the stub cannot express. A back end that cannot call a signature reports a compile error that names it (BACKEND gap 22), never a link error.
Library resolution: the form of foreign "name" and of the per-platform file names (library "sqlite3", windows: ..., linux: ..., macos: ...), where the mapping lives and what happens when the library is missing at start or at first call. Resolves the conflict between CONCEPT ("a package that contains foreign declarations says so in project.trb") and PROJECT.md section 1 ("native and foreign ... a manifest switch would be a second source of truth"): decided here, with the working assumption that the capability is derived from source as archive.trb does today and project.trb carries only the name-to-file mapping; the record may change that with a reason.
Capability and sandbox: unchanged contract restated in one place (visible in source, shown by torb add and the lock, never granted by Sandbox, SCRIPTS.md table); fixes the stale rule number in OS.md section 8 ("the-sandbox.md, rule 7" is rule 8) if it is still wrong; says what --embed-vm binaries and torb test do with a foreign library.
Ownership, callbacks, errors: memory C allocated (shared type with Close, DESTRUCTORS.md), what a pointer's lifetime means across a Task boundary (CONCURRENCY.md), the capture-free closure as function pointer (how it is lowered and kept alive), errno/GetLastError and a null result as Result, and a panic or a signal inside C. Platform arms (match OperatingSystem.current, OS.md) and which declarations are emitted per target.
What std/gpu and std/render need from the record (Layer 0 of COMPUTE.md): struct by value, out-parameters, function-pointer tables, pinned buffer windows, library loading at run time; stated as constraints, without designing those packages.
Slices: the record lists them in order (e.g. Pointer/CString in std; the library block and foreign type in parser and checker; the call in the VM; the call in the C back end; conformance tests against a tiny C test library built in CI), each one PR of one or two Conventional Commits, each with its seed impact under the two-commit rule (new syntax: teach both forms, refresh the seed, then use it; the seed refresh itself stays with the owner). A slice is filed as its own issue only when it starts.
Published packages: a section "Foreign functions in published packages" that sets out the options and the consequences for torb/sqlite (FIRST-PARTY.md row 7), the registry and torb publish, and is marked "open, owner decision (#84)". It carries no decision; a follow-up PR writes the answer into it, into RELEASE.md section 2 and into FIRST-PARTY.md.
Open design questions of the record are decided by the record (technical choices); one is left open only if it is a public commitment, a cost, a product direction or a breaking change of a published contract.
Gates green in both tiers.

Design record point

CONCEPT "Foreign Functions (Draft)" and "Capabilities are visible"; COMPUTE.md sections 1 (layer 0) and 2; NATIVE-BODIES.md (the other half of the native/foreign split); RELEASE.md sections 2, 7.8 and 7.11; PROJECT.md section 1 (what does not belong in a project file); SCRIPTS.md (capability table); VM.md section 6; BACKEND gap 22; DESTRUCTORS.md; JAVASCRIPT-AND-PHP.md (the JS boundary of foreign); OS.md (platform arms).

Seed impact

None for this step (documentation only; no std/, runtime/ or compiler/ file changes). The later slices add syntax and names the compiler looks up, so each states its seed impact under the two-commit rule.

Size

One PR, one Conventional Commit (docs(language): design record for foreign functions).

Out of scope

Any change to std/, runtime/ or compiler/; the decision on foreign in published packages (#84); std/gpu itself (#85); the seed.

Dependencies

None hard. Verified against main at c4a8443: no open or closed issue covers foreign functions besides #84; docs/design/NATIVE-BODIES.md is already on main (it is referenced, not a prerequisite). Part of #84; blocks #85.

## Context Step 1 of #84 (language: the design of foreign functions; COMPUTE.md section 2, ROADMAP "Computation and machine learning", CONCEPT "Foreign Functions (Draft)"). `foreign` is parsed and its block shape is checked (`checkForeignShape`, `compiler/src/semantics/checker/declaration.trb`), the package summary already reports it as the capability "foreign functions" (`compiler/src/package/archive.trb`), and the sandbox never grants it (`docs/language/configuration/the-sandbox.md`, rule 8). But nothing links or calls a foreign function, `Pointer<Value>` and `CString` are declared nowhere in `std/`, and no design record exists: CONCEPT is a draft with eight bullets, `docs/language/extensibility/foreign-functions.md` repeats them as "planned", and `std/gpu` ([COMPUTE.md](https://git.torb.dev/torbscript/language/src/branch/main/docs/design/COMPUTE.md) section 2: "then the design of foreign functions; then `std/gpu`") waits for it. This step is documentation only. ## Goal A design record `docs/design/FOREIGN.md` that specifies foreign functions end to end (types, ABI mapping, how each back end calls, library resolution, ownership, callbacks, errors) and cuts the build into slices, each one PR. The one question the record cannot answer itself, whether a published package may declare `foreign` functions and carry C sources, is an owner decision tracked in #84; the record states the options and leaves that section marked open. ## Acceptance criteria - [ ] `docs/design/FOREIGN.md` exists, headed and indexed like the other records (status line "designed, not built", an entry in `docs/design/index.md`); CONCEPT "Foreign Functions (Draft)", `docs/language/extensibility/foreign-functions.md`, COMPUTE.md (layer 0 and section 2), RELEASE.md section 2 and FIRST-PARTY.md row 7 link it; `docs/` regenerates `skills/` (`torb docs skill docs skills`, a tier A gate). - [ ] **Types and ABI:** a table from each foreign type to its C type and to the IR type it lowers to: the sized numbers, `Bool`, `Pointer<Value>`, `CString`, `foreign type` (C layout: field order, alignment, padding, no generics), and what is refused (`String`, `Int` without a size, closures other than the callback form, generics). Says in which `std` package `Pointer` and `CString` are declared, who may construct and dereference a pointer, how a `String` becomes a `CString` (copy, NUL check, lifetime of the call) and how a `Buffer<Item>` window is handed to C for the duration of one call. Each is a numbered decision with its reason. - [ ] **How the back ends call:** the C back end and the VM, both reading the same IR (VM.md section 6, BACKEND). Compares at least: run-time loading (`dlopen`/`LoadLibraryW`, the pattern `runtime/os/windows.c` and `runtime/platform.c` already use) against link-time linking (CONCEPT's "the compiler links directly"), and, for the VM's "generic trampoline", an in-tree call stub per ABI (SysV x86-64, Win64, AAPCS64) against libffi against generating a C stub per signature. Decides one, with the number of supported tier 1 targets, the dependency and licence cost (RELEASE.md section 7.11: no large C files in every build), and what happens on a signature the stub cannot express. A back end that cannot call a signature reports a compile error that names it (BACKEND gap 22), never a link error. - [ ] **Library resolution:** the form of `foreign "name"` and of the per-platform file names (`library "sqlite3", windows: ..., linux: ..., macos: ...`), where the mapping lives and what happens when the library is missing at start or at first call. Resolves the conflict between CONCEPT ("a package that contains `foreign` declarations says so in `project.trb`") and PROJECT.md section 1 ("`native` and `foreign` ... a manifest switch would be a second source of truth"): decided here, with the working assumption that the capability is derived from source as `archive.trb` does today and `project.trb` carries only the name-to-file mapping; the record may change that with a reason. - [ ] **Capability and sandbox:** unchanged contract restated in one place (visible in source, shown by `torb add` and the lock, never granted by `Sandbox`, SCRIPTS.md table); fixes the stale rule number in OS.md section 8 ("the-sandbox.md, rule 7" is rule 8) if it is still wrong; says what `--embed-vm` binaries and `torb test` do with a foreign library. - [ ] **Ownership, callbacks, errors:** memory C allocated (`shared type` with `Close`, DESTRUCTORS.md), what a pointer's lifetime means across a `Task` boundary (CONCURRENCY.md), the capture-free closure as function pointer (how it is lowered and kept alive), `errno`/`GetLastError` and a null result as `Result`, and a panic or a signal inside C. Platform arms (`match OperatingSystem.current`, OS.md) and which declarations are emitted per target. - [ ] **What `std/gpu` and `std/render` need** from the record (Layer 0 of COMPUTE.md): struct by value, out-parameters, function-pointer tables, pinned buffer windows, library loading at run time; stated as constraints, without designing those packages. - [ ] **Slices:** the record lists them in order (e.g. `Pointer`/`CString` in `std`; the `library` block and `foreign type` in parser and checker; the call in the VM; the call in the C back end; conformance tests against a tiny C test library built in CI), each one PR of one or two Conventional Commits, each with its seed impact under the two-commit rule (new syntax: teach both forms, refresh the seed, then use it; the seed refresh itself stays with the owner). A slice is filed as its own issue only when it starts. - [ ] **Published packages:** a section "Foreign functions in published packages" that sets out the options and the consequences for `torb/sqlite` (FIRST-PARTY.md row 7), the registry and `torb publish`, and is marked "open, owner decision (#84)". It carries no decision; a follow-up PR writes the answer into it, into RELEASE.md section 2 and into FIRST-PARTY.md. - [ ] Open design questions of the record are decided by the record (technical choices); one is left open only if it is a public commitment, a cost, a product direction or a breaking change of a published contract. - [ ] Gates green in both tiers. ## Design record point CONCEPT "Foreign Functions (Draft)" and "Capabilities are visible"; COMPUTE.md sections 1 (layer 0) and 2; NATIVE-BODIES.md (the other half of the `native`/`foreign` split); RELEASE.md sections 2, 7.8 and 7.11; PROJECT.md section 1 (what does not belong in a project file); SCRIPTS.md (capability table); VM.md section 6; BACKEND gap 22; DESTRUCTORS.md; JAVASCRIPT-AND-PHP.md (the JS boundary of `foreign`); OS.md (platform arms). ## Seed impact None for this step (documentation only; no `std/`, `runtime/` or `compiler/` file changes). The later slices add syntax and names the compiler looks up, so each states its seed impact under the two-commit rule. ## Size One PR, one Conventional Commit (`docs(language): design record for foreign functions`). ## Out of scope Any change to `std/`, `runtime/` or `compiler/`; the decision on `foreign` in published packages (#84); `std/gpu` itself (#85); the seed. ## Dependencies None hard. Verified against `main` at c4a8443: no open or closed issue covers foreign functions besides #84; `docs/design/NATIVE-BODIES.md` is already on `main` (it is referenced, not a prerequisite). Part of #84; blocks #85.

orion added the

loop

ready

labels

2026-10-24 02:16:17 +00:00

orion referenced this issue

2026-10-24 02:16:29 +00:00

language: the design of foreign functions #84

orion referenced this issue

2026-10-24 02:16:48 +00:00

language: the design of foreign functions #84

orion commented

2026-10-24 02:24:35 +00:00

Author

Collaborator

[loop:torbscript] claim implement by session:loop-torbscript-20261024 until 2026-10-24T04:24:35Z

orion commented

2026-10-24 02:25:44 +00:00

Author

Collaborator

[loop:torbscript] implement: checkpoint (context guard), no PR yet, nothing committed.

Done: worktree torbscript-language-459-foreign-design-record on branch loop/459-foreign-design-record (from main d5227eec, not pushed). Research read so far: COMPUTE.md, NATIVE-BODIES.md, CONCEPT "Foreign Functions (Draft)", docs/language/extensibility/foreign-functions.md, checkForeignShape and archive.trb (capability "foreign functions" derived from source), VM.md section 6 (the VM reaches C only through the kernel natives and a table of thunks generated from the manifest, so a foreign call needs one more kernel operation) and 10, BACKEND 3.7 / gap 22, dlopen-style loaders in runtime/platform.c and runtime/os/windows.c (only LoadLibraryW/GetProcAddress so far, no POSIX dlopen yet).
Facts for the record: docs/ is excluded from pdisc (.pdisc.yaml), so docs/design/*.md carry no frontmatter and need no stamp (still run pdisc.py check); design records are skill: omit, so a link to FOREIGN.md from docs/language/extensibility/foreign-functions.md keeps only its text in skills/ (hand-mirror the edit there, the copy differs only by one trailing blank line; add the record to source: in both). Entry in docs/design/index.md goes in the documents: list and in the generated block. Precedent commit: 798f20d9 (NATIVE-BODIES).
In flight: writing docs/design/FOREIGN.md.
Next: still to read OS.md sections 8 and 340-370, RELEASE.md sections 2/7.8/7.11, PROJECT.md section 1 (line 467, 1265), SCRIPTS.md table (line 163), DESTRUCTORS.md, CONCURRENCY.md (pointers across Task), JAVASCRIPT-AND-PHP.md (lines 80, 164), FIRST-PARTY.md row 7; then write the record, link it from CONCEPT, foreign-functions.md (+ skills copy), COMPUTE.md, RELEASE.md section 2, FIRST-PARTY.md row 7, fix the OS.md "rule 7" -> rule 8, open the PR for #459 (no Closes, #84 stays open), comment, remove the worktree.

[loop:torbscript] implement: checkpoint (context guard), no PR yet, nothing committed. - Done: worktree `torbscript-language-459-foreign-design-record` on branch `loop/459-foreign-design-record` (from main d5227eec, not pushed). Research read so far: COMPUTE.md, NATIVE-BODIES.md, CONCEPT "Foreign Functions (Draft)", docs/language/extensibility/foreign-functions.md, `checkForeignShape` and `archive.trb` (capability "foreign functions" derived from source), VM.md section 6 (the VM reaches C only through the kernel natives and a table of thunks generated from the manifest, so a foreign call needs one more kernel operation) and 10, BACKEND 3.7 / gap 22, dlopen-style loaders in `runtime/platform.c` and `runtime/os/windows.c` (only `LoadLibraryW`/`GetProcAddress` so far, no POSIX `dlopen` yet). - Facts for the record: `docs/` is excluded from pdisc (`.pdisc.yaml`), so docs/design/*.md carry no frontmatter and need no stamp (still run `pdisc.py check`); design records are `skill: omit`, so a link to FOREIGN.md from `docs/language/extensibility/foreign-functions.md` keeps only its text in `skills/` (hand-mirror the edit there, the copy differs only by one trailing blank line; add the record to `source:` in both). Entry in `docs/design/index.md` goes in the `documents:` list and in the generated block. Precedent commit: 798f20d9 (NATIVE-BODIES). - In flight: writing `docs/design/FOREIGN.md`. - Next: still to read OS.md sections 8 and 340-370, RELEASE.md sections 2/7.8/7.11, PROJECT.md section 1 (line 467, 1265), SCRIPTS.md table (line 163), DESTRUCTORS.md, CONCURRENCY.md (pointers across Task), JAVASCRIPT-AND-PHP.md (lines 80, 164), FIRST-PARTY.md row 7; then write the record, link it from CONCEPT, foreign-functions.md (+ skills copy), COMPUTE.md, RELEASE.md section 2, FIRST-PARTY.md row 7, fix the OS.md "rule 7" -> rule 8, open the PR for #459 (no `Closes`, #84 stays open), comment, remove the worktree.

orion commented

2026-10-24 02:29:10 +00:00

Author

Collaborator

[loop:torbscript] implement: checkpoint 2 (context guard), no PR yet.

Done: docs/design/FOREIGN.md written and committed locally (branch loop/459-foreign-design-record, worktree torbscript-language-459-foreign-design-record, NOT pushed). 11 sections: decisions, what exists, types/ABI, declarations + run-time loading in both back ends, checker rules F1-F7, sandbox/registry (yes to declaring, no to carrying C), C back end vs VM ForeignCall kernel op (stubs, spike gate vs libffi), memory/callbacks, non-goals, 8 slices, open points.
Next, in order: (1) re-read FOREIGN.md once and verify its references exist (docs/tooling/package-capabilities.md, section numbers of VM/OS/CONCURRENCY/DESTRUCTORS, "BACKEND 5.4"); (2) links to FOREIGN.md from CONCEPT.md (Foreign Functions section; also change "the compiler links directly" to the run-time loading of FOREIGN.md 4.2), docs/language/extensibility/foreign-functions.md (+ add to source:) and its skills/ copy by hand (design records are skill: omit), COMPUTE.md (lines 20 and 49), RELEASE.md section 2 (row 108) + 7.2 line 942, FIRST-PARTY.md row 7 (line 171) and row 9 (line 324), docs/design/index.md (the documents: list and the generated block near line 120, precedent 798f20d9); (3) OS.md line 1581 "rule 7" -> "rule 8"; (4) python pdisc.py check; (5) commit, push, open PR referencing #459 (no Closes, #84 stays open); (6) final next: review comment on #459; (7) remove worktree.
Note: git.torb.dev API needs ~/.secrets/torbdev-token (forgejo-token is for git.dmb.zone).

[loop:torbscript] implement: checkpoint 2 (context guard), no PR yet. - Done: `docs/design/FOREIGN.md` written and committed locally (branch `loop/459-foreign-design-record`, worktree `torbscript-language-459-foreign-design-record`, NOT pushed). 11 sections: decisions, what exists, types/ABI, declarations + run-time loading in both back ends, checker rules F1-F7, sandbox/registry (yes to declaring, no to carrying C), C back end vs VM `ForeignCall` kernel op (stubs, spike gate vs libffi), memory/callbacks, non-goals, 8 slices, open points. - Next, in order: (1) re-read FOREIGN.md once and verify its references exist (`docs/tooling/package-capabilities.md`, section numbers of VM/OS/CONCURRENCY/DESTRUCTORS, "BACKEND 5.4"); (2) links to FOREIGN.md from CONCEPT.md (Foreign Functions section; also change "the compiler links directly" to the run-time loading of FOREIGN.md 4.2), `docs/language/extensibility/foreign-functions.md` (+ add to `source:`) and its `skills/` copy by hand (design records are `skill: omit`), COMPUTE.md (lines 20 and 49), RELEASE.md section 2 (row 108) + 7.2 line 942, FIRST-PARTY.md row 7 (line 171) and row 9 (line 324), `docs/design/index.md` (the `documents:` list and the generated block near line 120, precedent 798f20d9); (3) OS.md line 1581 "rule 7" -> "rule 8"; (4) `python pdisc.py check`; (5) commit, push, open PR referencing #459 (no Closes, #84 stays open); (6) final `next: review` comment on #459; (7) remove worktree. - Note: git.torb.dev API needs `~/.secrets/torbdev-token` (forgejo-token is for git.dmb.zone).

orion commented

2026-10-24 02:29:27 +00:00

Author

Collaborator

[loop:torbscript] claim implement by session:loop-torbscript-20261024 until 2026-10-24T04:29:27Z

~~orion referenced this issue 2026-10-24 02:31:11 +00:00~~

docs(foreign): design record for foreign functions #460

orion referenced this issue from a pull request that will close it,

2026-10-24 02:31:17 +00:00

docs(foreign): design record for foreign functions #460

orion commented

2026-10-24 02:31:17 +00:00

Author

Collaborator

[loop:torbscript] implement: design record docs/design/FOREIGN.md plus cross-links (CONCEPT, language page + skills copy, COMPUTE, RELEASE, FIRST-PARTY, design index), OS.md rule 7->8 fix, pdisc clean; PR #460 (docs only, CI is the gate), next: review

[loop:torbscript] implement: design record docs/design/FOREIGN.md plus cross-links (CONCEPT, language page + skills copy, COMPUTE, RELEASE, FIRST-PARTY, design index), OS.md rule 7->8 fix, pdisc clean; PR https://git.torb.dev/torbscript/language/pulls/460 (docs only, CI is the gate), next: review