torbscript/language
Watch
1
0
Fork
You've already forked language
 0
Code Issues 139 Pull requests 2 Projects Releases 18 Packages 3 Wiki Activity Actions

resources: embedding (RESOURCES slice 2) #135

New issue
Closed
opened 2026-10-05 21:07:24 +00:00 by orion · 20 comments
orion commented 2026-10-05 21:07:24 +00:00
Collaborator
Copy link

Context

EmbeddedBytes.bytes() and EmbeddedText.text() as static data in the C back end.

Source

Acceptance criteria

  • As RESOURCES.md slice 2 specifies

Dependencies

None known.

Moved from the planning lists of the docs; the design record stays the specification.

## Context `EmbeddedBytes.bytes()` and `EmbeddedText.text()` as static data in the C back end. ## Source - [RESOURCES.md slice 2](https://git.torb.dev/torbscript/language/src/commit/393ef2608e4a60b8ff76d8b6b408951b822d887d/docs/design/RESOURCES.md#L742) ## Acceptance criteria - [ ] As RESOURCES.md slice 2 specifies ## Dependencies None known. _Moved from the planning lists of the docs; the design record stays the specification._
orion commented 2026-10-09 05:42:18 +00:00
Author
Collaborator
Copy link

[loop:torbscript] intake: in scope (author orion, label loop), source RESOURCES.md slice 2, not a duplicate. C back end change, likely natives (two-commit rule at refine). No dependencies. next: waiting (refine queued, ready buffer full) until 2026-10-10T04:00:00Z

[loop:torbscript] intake: in scope (author orion, label loop), source RESOURCES.md slice 2, not a duplicate. C back end change, likely natives (two-commit rule at refine). No dependencies. next: waiting (refine queued, ready buffer full) until 2026-10-10T04:00:00Z
orion commented 2026-10-25 01:10:51 +00:00
Author
Collaborator
Copy link

[loop:torbscript] claim refine by session:loop-torbscript-20261025-0258 until 2026-10-25T03:15:00Z

[loop:torbscript] claim refine by session:loop-torbscript-20261025-0258 until 2026-10-25T03:15:00Z
orion commented 2026-10-25 01:11:46 +00:00
Author
Collaborator
Copy link

[loop:torbscript] decided by the loop: scope, mechanism and size of RESOURCES slice 2, from docs/design/RESOURCES.md section 10 (slice 2), section 9 (C back end) and probe 2, and the current main (std/resource is slice 1: three capsules over the stable name, no native members yet).

Goal. A program can call EmbeddedBytes.bytes(): Bytes and EmbeddedText.text(): String on a value made from a literal (SpriteSheet.embedded("./hero.png") style) and gets the file's content as static data of the generated C, synchronous and infallible.

Acceptance criteria (each testable by a conformance program with a .expected, byte-identical on both implementations, or by a checker test in compiler/tests/)

  1. EmbeddedText.text() returns exactly the file's text; a multi-byte (UTF-8) file and an empty file are covered.
  2. EmbeddedBytes.bytes() returns exactly the file's bytes, including a file that is not valid UTF-8 and a file containing 0x00 and 0xFF; length and a few indexed bytes are printed.
  3. A literal where an EmbeddedText is expected naming a file that is not valid UTF-8 is a diagnostic at the literal's line (error, names the file's stable name and the offset of the first bad byte). The same file as EmbeddedBytes is fine. No replacement character is ever produced.
  4. The content is static data in the generated C (the shape of probe 2: immortal storage plus a const torb_text/torb_bytes header), not a call at first read; the same stable name used at several call sites (also from EmbeddedBytes and EmbeddedText both) is emitted once.
  5. name() and show() still print the stable name (slice 1 behaviour unchanged; the existing literals.test.trb stays green).
  6. Out of scope, stays as is: shipping Resource (slice 3), the embedded total / resources { } budget (slice 4), VM behaviour, folding through static initializers (slice 6). No size limit in this slice.

Technical decisions (loop's, per the contract's "internal and technical design choices")

  • The checker reads the file where the literal is resolved (it already lists the directory for the case check), validates UTF-8 for EmbeddedText, and records the content with the resolved name in the IR next to the stable name; the lowering of the literal keeps building the existing capsule value (Self nameValue), so Equals/Hash/Show stay on the name and nothing in slice 1's contract changes.
  • The C back end emits one static data block per distinct stable name (bytes and text share storage when both kinds name the same file) and the two natives are a lookup of that block by the capsule's name, resolved inside the generated program (a generated function over the names the program embeds). This is an internal lookup, not a user-visible by-string API; the design record's "no run-time lookup by string" is about the language surface. If the implementer finds the capsule can carry an index without breaking slice 1's Equals/Hash, that is allowed, as long as criteria 4 and 5 hold.
  • Binary content is emitted as the byte list the emitter already writes for string literals (probe 2, about 6 bytes of C per payload byte, no limit needed below 16 MB per probe 3).

Design record point. docs/design/RESOURCES.md section 10, slice 2 (and section 9 "The C back end"); the doc's status line and the slice table row get "Done" with the PR date, docs/standard-library/resource.md is slice 1's follow-up and is only touched for the two members.

Seed. Yes, touches the seed: two new natives (EmbeddedBytes.bytes, EmbeddedText.text) in compiler/src/backend/c/natives.trb. Two-commit rule per CLAUDE.md: (1) feat(backend): embed resource files as static data teaches the compiler the natives, the content record in the IR, the UTF-8 check and the emission, with checker/IR tests and no native fn in std/resource yet; (2) feat(resource): read embedded files adds the two native fn members to std/resource, the conformance programs and the skills/ regeneration. Commit 2 needs a published seed that already knows commit 1: the PR waits for Torben's seed refresh/publish after commit 1 is on main (the seed itself stays Torben's, forbidden to the loop), so the implementer opens the PR with commit 1 only if commit 2 cannot bootstrap, and commit 2 follows as the same issue's second PR. Size: two Conventional Commits, one issue.

[loop:torbscript] decided by the loop: scope, mechanism and size of RESOURCES slice 2, from `docs/design/RESOURCES.md` section 10 (slice 2), section 9 (C back end) and probe 2, and the current `main` (`std/resource` is slice 1: three capsules over the stable name, no native members yet). **Goal.** A program can call `EmbeddedBytes.bytes(): Bytes` and `EmbeddedText.text(): String` on a value made from a literal (`SpriteSheet.embedded("./hero.png")` style) and gets the file's content as static data of the generated C, synchronous and infallible. **Acceptance criteria** (each testable by a conformance program with a `.expected`, byte-identical on both implementations, or by a checker test in `compiler/tests/`) 1. `EmbeddedText.text()` returns exactly the file's text; a multi-byte (UTF-8) file and an empty file are covered. 2. `EmbeddedBytes.bytes()` returns exactly the file's bytes, including a file that is not valid UTF-8 and a file containing `0x00` and `0xFF`; length and a few indexed bytes are printed. 3. A literal where an `EmbeddedText` is expected naming a file that is not valid UTF-8 is a diagnostic at the literal's line (error, names the file's stable name and the offset of the first bad byte). The same file as `EmbeddedBytes` is fine. No replacement character is ever produced. 4. The content is static data in the generated C (the shape of probe 2: immortal storage plus a `const torb_text`/`torb_bytes` header), not a call at first read; the same stable name used at several call sites (also from `EmbeddedBytes` and `EmbeddedText` both) is emitted once. 5. `name()` and `show()` still print the stable name (slice 1 behaviour unchanged; the existing `literals.test.trb` stays green). 6. Out of scope, stays as is: shipping `Resource` (slice 3), the embedded total / `resources { }` budget (slice 4), VM behaviour, folding through `static` initializers (slice 6). No size limit in this slice. **Technical decisions** (loop's, per the contract's "internal and technical design choices") - The checker reads the file where the literal is resolved (it already lists the directory for the case check), validates UTF-8 for `EmbeddedText`, and records the content with the resolved name in the IR next to the stable name; the lowering of the literal keeps building the existing capsule value (`Self nameValue`), so `Equals`/`Hash`/`Show` stay on the name and nothing in slice 1's contract changes. - The C back end emits one static data block per distinct stable name (bytes and text share storage when both kinds name the same file) and the two natives are a lookup of that block by the capsule's name, resolved inside the generated program (a generated function over the names the program embeds). This is an internal lookup, not a user-visible by-string API; the design record's "no run-time lookup by string" is about the language surface. If the implementer finds the capsule can carry an index without breaking slice 1's `Equals`/`Hash`, that is allowed, as long as criteria 4 and 5 hold. - Binary content is emitted as the byte list the emitter already writes for string literals (probe 2, about 6 bytes of C per payload byte, no limit needed below 16 MB per probe 3). **Design record point.** `docs/design/RESOURCES.md` section 10, slice 2 (and section 9 "The C back end"); the doc's status line and the slice table row get "Done" with the PR date, `docs/standard-library/resource.md` is slice 1's follow-up and is only touched for the two members. **Seed.** Yes, touches the seed: two new natives (`EmbeddedBytes.bytes`, `EmbeddedText.text`) in `compiler/src/backend/c/natives.trb`. Two-commit rule per `CLAUDE.md`: (1) `feat(backend): embed resource files as static data` teaches the compiler the natives, the content record in the IR, the UTF-8 check and the emission, with checker/IR tests and no `native fn` in `std/resource` yet; (2) `feat(resource): read embedded files` adds the two `native fn` members to `std/resource`, the conformance programs and the `skills/` regeneration. Commit 2 needs a published seed that already knows commit 1: the PR waits for Torben's seed refresh/publish after commit 1 is on `main` (the seed itself stays Torben's, forbidden to the loop), so the implementer opens the PR with commit 1 only if commit 2 cannot bootstrap, and commit 2 follows as the same issue's second PR. Size: two Conventional Commits, one issue.
orion commented 2026-10-25 01:11:46 +00:00
Author
Collaborator
Copy link

[loop:torbscript] refine: #135 brought to the definition of ready (goal, six testable acceptance criteria, design point RESOURCES.md section 10 slice 2, seed statement: touches the seed via two new natives, two-commit rule; size two commits). Decisions recorded in the comment above. Label ready set. Motivation: unblocks the native benchmark number #93 waits on. next: ready (implement; commit 2 needs a seed refreshed by Torben after commit 1). Tokens: about 60k (RE worker, sonnet).

[loop:torbscript] refine: #135 brought to the definition of ready (goal, six testable acceptance criteria, design point RESOURCES.md section 10 slice 2, seed statement: touches the seed via two new natives, two-commit rule; size two commits). Decisions recorded in the comment above. Label `ready` set. Motivation: unblocks the native benchmark number #93 waits on. next: ready (implement; commit 2 needs a seed refreshed by Torben after commit 1). Tokens: about 60k (RE worker, sonnet).
orion commented 2026-10-25 01:20:49 +00:00
Author
Collaborator
Copy link

[loop:torbscript] claim implement by session:loop-torbscript-20261025-0258 until 2026-10-25T07:25:00Z

[loop:torbscript] claim implement by session:loop-torbscript-20261025-0258 until 2026-10-25T07:25:00Z
orion commented 2026-10-25 01:22:15 +00:00
Author
Collaborator
Copy link

[loop:torbscript] implement: checkpoint, worker context budget reached before any code was written. Done: worktree code/wt/torbscript-language-135 (workspace orion) on branch loop/135-resource-embedding from main 32d880c9, nothing committed or pushed yet; contract, CLAUDE.md, CONTRIBUTING.md and RESOURCES.md sections 4, 6, 9, 10 read. Scope for this PR is commit 1 only (feat(backend): embed resource files as static data: IR content record, UTF-8 check in the checker, C emission, the natives in compiler/src/backend/c/natives.trb, checker/IR tests, no native fn in std/resource), opened with Refs #135; commit 2 follows as a second PR after the seed refresh. Entry points found: compiler/src/semantics/checker/literal.trb (slice 1 literal rule), checker/wellknown.trb, checker/context.trb, compiler/src/ir/lower/literal.trb, compiler/tests/literals.test.trb, std/resource/src/resource.trb. Next: read these plus the C emitter string-literal storage path, implement, torb format --check on touched files, push, PR, request review from argus. next: implement (continuation worker). Tokens: about 85k (sonnet).

[loop:torbscript] implement: checkpoint, worker context budget reached before any code was written. Done: worktree `code/wt/torbscript-language-135` (workspace `orion`) on branch `loop/135-resource-embedding` from `main` 32d880c9, nothing committed or pushed yet; contract, CLAUDE.md, CONTRIBUTING.md and RESOURCES.md sections 4, 6, 9, 10 read. Scope for this PR is commit 1 only (`feat(backend): embed resource files as static data`: IR content record, UTF-8 check in the checker, C emission, the natives in `compiler/src/backend/c/natives.trb`, checker/IR tests, no `native fn` in `std/resource`), opened with `Refs #135`; commit 2 follows as a second PR after the seed refresh. Entry points found: `compiler/src/semantics/checker/literal.trb` (slice 1 literal rule), `checker/wellknown.trb`, `checker/context.trb`, `compiler/src/ir/lower/literal.trb`, `compiler/tests/literals.test.trb`, `std/resource/src/resource.trb`. Next: read these plus the C emitter string-literal storage path, implement, `torb format --check` on touched files, push, PR, request review from `argus`. next: implement (continuation worker). Tokens: about 85k (sonnet).
orion commented 2026-10-25 01:34:04 +00:00
Author
Collaborator
Copy link

[loop:torbscript] implement: checkpoint 2 (continuation 1), context guard hit after reading, no code pushed yet (branch loop/135-resource-embedding = main 32d880c9, worktree code/wt/torbscript-language-135). Design found, to implement as commit 1 without re-reading:

  1. Bytes source. compiler/src/project/source-tree.trb: SourceTree holds texts only (files: Map<String,String>), non-source files are only listed (otherPaths, set by project/read.trb listBelow). Add fn resourceBytes(path): Bytes? (in-memory files entry -> its text as UTF-8 bytes; otherPaths entry -> File.readBytes(path).ok(); Bytes is std/stream Buffer<UInt8>). Workspace (project/workspace.trb, built in one place: semantics/check.trb:147 Workspace packages, listedFiles: tree.listedPaths()) gets a closure field readResource: (String) -> Bytes? defaulting to { None }, set from the tree there.
  2. Checker. semantics/checker/literal.trb checkedTextLiteral, resource branch (resourceNameOf returns the stable name; its local path is the listed key to read): for .EmbeddedBytesLiteral/.EmbeddedTextLiteral read the bytes, for text validate UTF-8 (diagnostic at the literal: stable name + offset of first bad byte, criterion 3), and record. Add content: Bytes? = None to CheckedLiteral (checker/context.trb:386); text stays the stable name, so lower/literal.trb (literalCellOf, builds the capsule via literal(text)) is unchanged.
  3. Lowering/natives. Template is ir/lower/script.trb scriptTableOf (a generated body: chain of IntrinsicOperation.TextEqual over constants, nativeWrapperName, lowering.nativeWrappers cache) dispatched at ir/lower/call.trb:865 (.ScriptTable(kind)), listed in ir/witness.trb:489 and call.trb:2242; manifest rows at backend/c/natives.trb:1982 (NativeTarget.ScriptTable). Plan: new NativeTarget.Embedded(kind: EmbeddedKind {Bytes, Text}), manifest rows embeddedBytes/embeddedText (free natives taking the stable name; commit 2 wraps them: fn bytes(): Bytes { embeddedBytes(nameValue) }, avoids any capsule-layout question). New ir/lower/embedded.trb: body = for each distinct stable name (sorted) with content over all modules' checkedLiterals (one entry per name; text-kind and bytes-kind share), compare slot 0 with the name constant, return a static. Text result: StaticContents.Literal(LiteralValue.Text(..)) + Instruction.Constant already works (see textStatic in script.trb). Bytes result: no bytes static exists yet; add StaticContents.Blob(bytes)-style case (or emit the Bytes as immortal torb_bytes + slice like emitTextStorage in backend/c/emit.trb:2237; Bytes = Buffer<UInt8>, check its IR type/layout before choosing). Every match over StaticContents/NativeTarget needs the new case (grep both).
  4. Tests (commit 1, checker/IR only, no native fn in std/resource yet): compiler/tests/literals.test.trb (UTF-8 diagnostic for EmbeddedText; bytes file ok), an IR/lowering test for the table (dedup per stable name). Check whether a test requires every manifest entry to have a native declaration in std/ (grep natives in compiler/tests/): if so the rows must be .Planned-free but declared in commit 2 only; adapt.
  5. Docs: RESOURCES.md status line + slice 2 row "Done" only when commit 2 lands.

Pushed: nothing. Remaining: everything above. next: implement (continuation 2). Tokens: about 85k (sonnet).

[loop:torbscript] implement: checkpoint 2 (continuation 1), context guard hit after reading, no code pushed yet (branch `loop/135-resource-embedding` = main 32d880c9, worktree `code/wt/torbscript-language-135`). Design found, to implement as commit 1 without re-reading: 1. **Bytes source.** `compiler/src/project/source-tree.trb`: `SourceTree` holds texts only (`files: Map<String,String>`), non-source files are only listed (`otherPaths`, set by `project/read.trb` `listBelow`). Add `fn resourceBytes(path): Bytes?` (in-memory `files` entry -> its text as UTF-8 bytes; `otherPaths` entry -> `File.readBytes(path).ok()`; `Bytes` is `std/stream` `Buffer<UInt8>`). `Workspace` (`project/workspace.trb`, built in one place: `semantics/check.trb:147` `Workspace packages, listedFiles: tree.listedPaths()`) gets a closure field `readResource: (String) -> Bytes?` defaulting to `{ None }`, set from the tree there. 2. **Checker.** `semantics/checker/literal.trb` `checkedTextLiteral`, resource branch (`resourceNameOf` returns the stable name; its local `path` is the listed key to read): for `.EmbeddedBytesLiteral`/`.EmbeddedTextLiteral` read the bytes, for text validate UTF-8 (diagnostic at the literal: stable name + offset of first bad byte, criterion 3), and record. Add `content: Bytes? = None` to `CheckedLiteral` (`checker/context.trb:386`); `text` stays the stable name, so `lower/literal.trb` (`literalCellOf`, builds the capsule via `literal(text)`) is unchanged. 3. **Lowering/natives.** Template is `ir/lower/script.trb` `scriptTableOf` (a generated body: chain of `IntrinsicOperation.TextEqual` over constants, `nativeWrapperName`, `lowering.nativeWrappers` cache) dispatched at `ir/lower/call.trb:865` (`.ScriptTable(kind)`), listed in `ir/witness.trb:489` and call.trb:2242; manifest rows at `backend/c/natives.trb:1982` (`NativeTarget.ScriptTable`). Plan: new `NativeTarget.Embedded(kind: EmbeddedKind {Bytes, Text})`, manifest rows `embeddedBytes`/`embeddedText` (free natives taking the stable name; commit 2 wraps them: `fn bytes(): Bytes { embeddedBytes(nameValue) }`, avoids any capsule-layout question). New `ir/lower/embedded.trb`: body = for each distinct stable name (sorted) with `content` over all modules' `checkedLiterals` (one entry per name; text-kind and bytes-kind share), compare slot 0 with the name constant, return a static. Text result: `StaticContents.Literal(LiteralValue.Text(..))` + `Instruction.Constant` already works (see `textStatic` in script.trb). Bytes result: no bytes static exists yet; add `StaticContents.Blob(bytes)`-style case (or emit the Bytes as immortal `torb_bytes` + slice like `emitTextStorage` in `backend/c/emit.trb:2237`; `Bytes` = `Buffer<UInt8>`, check its IR type/layout before choosing). Every `match` over `StaticContents`/`NativeTarget` needs the new case (grep both). 4. Tests (commit 1, checker/IR only, no `native fn` in `std/resource` yet): `compiler/tests/literals.test.trb` (UTF-8 diagnostic for EmbeddedText; bytes file ok), an IR/lowering test for the table (dedup per stable name). Check whether a test requires every manifest entry to have a `native` declaration in `std/` (grep `natives` in `compiler/tests/`): if so the rows must be `.Planned`-free but declared in commit 2 only; adapt. 5. Docs: RESOURCES.md status line + slice 2 row "Done" only when commit 2 lands. Pushed: nothing. Remaining: everything above. next: implement (continuation 2). Tokens: about 85k (sonnet).
orion commented 2026-10-25 01:44:42 +00:00
Author
Collaborator
Copy link

[loop:torbscript] implement: checkpoint 3 (continuation 2), context guard hit. Pushed on loop/135-resource-embedding (head f6180358, worktree code/wt/torbscript-language-135, no PR yet):

  • ca9408b4 feat(project): read the bytes of a listed resource file: SourceTree.resourceBytes(path): Bytes? (source-tree.trb), Workspace.readResource: (String) => Bytes? (workspace.trb, default { _ => None }), set in semantics/check.trb (Workspace packages, listedFiles:, readResource:).
  • f6180358 feat(checker): read the bytes of an embedded resource literal: CheckedLiteral.content: Bytes? = None (checker/context.trb), resourceNameOf now returns (name, path)?, new embeddedContentOf in checker/literal.trb (unreadable file and non-UTF-8 EmbeddedText diagnostics with the byte offset, via textOf of std/stream).

Remaining for commit 1 (nothing of it written yet), decisions made:

  1. Bytes representation: no new StaticContents case. A Blob static would need C emit (backend/c/emit.trb:2221/2339), bytecode emit (backend/bytecode/emit.trb:626), vm/interpret.trb:1597, vm/embed.trb:204, ir/print.trb:549: too wide to write blind. Instead embed an EmbeddedBytes as the Base64 text (Base64Alphabet.Standard.encoded(bytes), use Base64Alphabet from "std/encoding") in an ordinary text static; commit 2 decodes it in std/resource (bytesOfBase64). EmbeddedText stays a plain text static. Record this deviation from "static data" in RESOURCES.md section 6 and as a follow-up (real Blob static).
  2. Checker (checker/context.trb, next to isReadVerbatim ~l.1651): add fn embeddedContents(): Map<String, Bytes>: for module in program.graph.modules, for (_, literal) in tables[module.id.value].checkedLiterals, if literal.content is Some, set literal.text, bytes.
  3. backend/c/natives.trb: case Embedded(kind: EmbeddedKind) in NativeTarget (~l.320 after ScriptTable), public type EmbeddedKind { case Bytes case Text }; rows after l.1995: NativeEntry(name: "embeddedBytes", target: NativeTarget.Embedded(EmbeddedKind.Bytes)) and embeddedText (free natives (name: String): String, declared in std/resource only in commit 2).
  4. New ir/lower/embedded.trb, modelled on ir/lower/script.trb scriptTableOf (l.30-110: nativeWrapperName, lowering.nativeWrappers cache, newFunction, chain of IntrinsicOperation.TextEqual on Slot(0) against textStatic constants, Terminator.Return(borrowed(..)), final fallthrough returns the empty text). One branch per distinct stable name, sorted; Text kind: textOf(bytes) Ok only; Bytes kind: Base64 text. Dispatch in ir/lower/call.trb next to .ScriptTable(kind) (l.865): .Embedded(kind) => { table = embeddedTableOf(...); emitCall lowering, builder, destination, table, operands, annotation, at }; add .Embedded(_) to call.trb:2242 (operator, unsupported), ir/witness.trb:489 (or-pattern), compiler/tests/natives-audit.test.trb:114 (kindOf: "Embedded").
  5. Docs/test for the audit (compiler/tests/natives-audit.test.trb checks the record): docs/design/NATIVES.md: Target list l.20 add Embedded; edges table l.66-78 add | resource | the files the compiler embeds into the program (std/resource) |; table: new section ### Embedded resources / Module: std/resource. before ## The drift check (l.870) with rows | \embeddedBytes` | Embedded | keep | resource | - | - | - |andembeddedText`; Counts l.97-105: keep 181->183, total 536->538.
  6. Tests: compiler/tests/literals.test.trb (non-UTF-8 EmbeddedText diagnostic, EmbeddedBytes of the same file ok, content recorded); an IR test for dedup per stable name if a harness exists nearby (grep scriptBody in compiler/tests found none, so checker-level test only).
  7. Then: push (use git -c "http.extraHeader=Authorization: token $(cat ~/.secrets/torbdev-token)" push origin loop/135-resource-embedding), open PR against main (Refs #135, mention the Base64 deviation + commit 2 pending Torben's seed refresh), request review from argus, wait for CI <= 5 min, write-back.

Unverified syntax guesses to watch in CI: closure field (String) => Bytes? with default { _ => None } in Workspace (public type, may need Send/Eq), Bytes prelude visibility in project/source-tree.trb and workspace.trb, Some((name, path)) tuple pattern in literal.trb:130, .bytes() on String.
Tokens: ~85k (sonnet). next: implement (continuation 3).

[loop:torbscript] implement: checkpoint 3 (continuation 2), context guard hit. Pushed on `loop/135-resource-embedding` (head f6180358, worktree `code/wt/torbscript-language-135`, no PR yet): - `ca9408b4 feat(project): read the bytes of a listed resource file`: `SourceTree.resourceBytes(path): Bytes?` (source-tree.trb), `Workspace.readResource: (String) => Bytes?` (workspace.trb, default `{ _ => None }`), set in `semantics/check.trb` (`Workspace packages, listedFiles:, readResource:`). - `f6180358 feat(checker): read the bytes of an embedded resource literal`: `CheckedLiteral.content: Bytes? = None` (checker/context.trb), `resourceNameOf` now returns `(name, path)?`, new `embeddedContentOf` in checker/literal.trb (unreadable file and non-UTF-8 `EmbeddedText` diagnostics with the byte offset, via `textOf` of std/stream). Remaining for commit 1 (nothing of it written yet), decisions made: 1. **Bytes representation: no new `StaticContents` case.** A Blob static would need C emit (`backend/c/emit.trb:2221/2339`), bytecode emit (`backend/bytecode/emit.trb:626`), `vm/interpret.trb:1597`, `vm/embed.trb:204`, `ir/print.trb:549`: too wide to write blind. Instead embed an EmbeddedBytes as the Base64 text (`Base64Alphabet.Standard.encoded(bytes)`, `use Base64Alphabet from "std/encoding"`) in an ordinary text static; commit 2 decodes it in `std/resource` (`bytesOfBase64`). EmbeddedText stays a plain text static. Record this deviation from "static data" in RESOURCES.md section 6 and as a follow-up (real Blob static). 2. `Checker` (checker/context.trb, next to `isReadVerbatim` ~l.1651): add `fn embeddedContents(): Map<String, Bytes>`: for `module in program.graph.modules`, for `(_, literal)` in `tables[module.id.value].checkedLiterals`, if `literal.content` is Some, `set literal.text, bytes`. 3. `backend/c/natives.trb`: `case Embedded(kind: EmbeddedKind)` in `NativeTarget` (~l.320 after `ScriptTable`), `public type EmbeddedKind { case Bytes case Text }`; rows after l.1995: `NativeEntry(name: "embeddedBytes", target: NativeTarget.Embedded(EmbeddedKind.Bytes))` and `embeddedText` (free natives `(name: String): String`, declared in std/resource only in commit 2). 4. New `ir/lower/embedded.trb`, modelled on `ir/lower/script.trb` `scriptTableOf` (l.30-110: `nativeWrapperName`, `lowering.nativeWrappers` cache, `newFunction`, chain of `IntrinsicOperation.TextEqual` on `Slot(0)` against `textStatic` constants, `Terminator.Return(borrowed(..))`, final fallthrough returns the empty text). One branch per distinct stable name, sorted; Text kind: `textOf(bytes)` Ok only; Bytes kind: Base64 text. Dispatch in `ir/lower/call.trb` next to `.ScriptTable(kind)` (l.865): `.Embedded(kind) => { table = embeddedTableOf(...); emitCall lowering, builder, destination, table, operands, annotation, at }`; add `.Embedded(_)` to `call.trb:2242` (operator, unsupported), `ir/witness.trb:489` (or-pattern), `compiler/tests/natives-audit.test.trb:114` (`kindOf`: `"Embedded"`). 5. Docs/test for the audit (`compiler/tests/natives-audit.test.trb` checks the record): `docs/design/NATIVES.md`: Target list l.20 add `Embedded`; edges table l.66-78 add `| resource | the files the compiler embeds into the program (`std/resource`) |`; table: new section `### Embedded resources` / `Module: std/resource.` before `## The drift check` (l.870) with rows `| \`embeddedBytes\` | Embedded | keep | resource | - | - | - |` and `embeddedText`; Counts l.97-105: keep 181->183, total 536->538. 6. Tests: `compiler/tests/literals.test.trb` (non-UTF-8 EmbeddedText diagnostic, EmbeddedBytes of the same file ok, content recorded); an IR test for dedup per stable name if a harness exists nearby (grep `scriptBody` in compiler/tests found none, so checker-level test only). 7. Then: push (use `git -c "http.extraHeader=Authorization: token $(cat ~/.secrets/torbdev-token)" push origin loop/135-resource-embedding`), open PR against main (`Refs #135`, mention the Base64 deviation + commit 2 pending Torben's seed refresh), request review from argus, wait for CI <= 5 min, write-back. Unverified syntax guesses to watch in CI: closure field `(String) => Bytes?` with default `{ _ => None }` in `Workspace` (public type, may need Send/Eq), `Bytes` prelude visibility in project/source-tree.trb and workspace.trb, `Some((name, path))` tuple pattern in literal.trb:130, `.bytes()` on String. Tokens: ~85k (sonnet). next: implement (continuation 3).
orion commented 2026-10-25 01:53:19 +00:00
Author
Collaborator
Copy link

[loop:torbscript] implement: commit 1 complete and pushed as PR #479 (head ec60a5e8, review requested from argus): feat(backend): embed resource files as static data (checker embeddedContents, native target Embedded with embeddedBytes/embeddedText, ir/lower/embedded.trb, NATIVES.md rows and counts, RESOURCES.md section 6 note) plus a literals test. EmbeddedBytes are stored as Base64 text (no blob static yet), recorded in RESOURCES.md. Nothing built locally; CI is the compiler: after ~5 min the bootstrap on the PR was still running (an earlier superseded run was cancelled), so unverified. Commit 2 (std/resource declarations, conformance programs) waits for the seed refresh by Torben. next: waiting (CI on PR #479). Tokens: ~75k (sonnet).

[loop:torbscript] implement: commit 1 complete and pushed as PR https://git.torb.dev/torbscript/language/pulls/479 (head ec60a5e8, review requested from argus): `feat(backend): embed resource files as static data` (checker `embeddedContents`, native target `Embedded` with `embeddedBytes`/`embeddedText`, `ir/lower/embedded.trb`, NATIVES.md rows and counts, RESOURCES.md section 6 note) plus a literals test. EmbeddedBytes are stored as Base64 text (no blob static yet), recorded in RESOURCES.md. Nothing built locally; CI is the compiler: after ~5 min the bootstrap on the PR was still running (an earlier superseded run was cancelled), so unverified. Commit 2 (std/resource declarations, conformance programs) waits for the seed refresh by Torben. next: waiting (CI on PR https://git.torb.dev/torbscript/language/pulls/479). Tokens: ~75k (sonnet).
orion commented 2026-10-25 01:58:12 +00:00
Author
Collaborator
Copy link

[loop:torbscript] implement: commit 1 of resource embedding is up as #479 (head e380a73f, review requested from argus): SourceTree.resourceBytes and Workspace.readResource, the checker reads the file at an EmbeddedBytes/EmbeddedText literal (CheckedLiteral.content, UTF-8 diagnostic with offset), NativeTarget.Embedded with embeddedBytes/embeddedText rows and ir/lower/embedded.trb (EmbeddedBytes as Base64 text, no Blob static yet), tests in literals.test.trb. This run also restored the CRLF line endings of context.trb that an earlier commit had converted (net diff there is +21 lines). Commit 2 (std/resource) waits for the seed refresh. next: waiting (CI on PR #479; commit messages, scope and pdisc green, gates pending after 5 min). Claim released. Tokens: about 60k (sonnet).

[loop:torbscript] implement: commit 1 of resource embedding is up as https://git.torb.dev/torbscript/language/pulls/479 (head e380a73f, review requested from argus): SourceTree.resourceBytes and Workspace.readResource, the checker reads the file at an EmbeddedBytes/EmbeddedText literal (CheckedLiteral.content, UTF-8 diagnostic with offset), NativeTarget.Embedded with embeddedBytes/embeddedText rows and ir/lower/embedded.trb (EmbeddedBytes as Base64 text, no Blob static yet), tests in literals.test.trb. This run also restored the CRLF line endings of context.trb that an earlier commit had converted (net diff there is +21 lines). Commit 2 (std/resource) waits for the seed refresh. next: waiting (CI on PR https://git.torb.dev/torbscript/language/pulls/479; commit messages, scope and pdisc green, gates pending after 5 min). Claim released. Tokens: about 60k (sonnet).
orion commented 2026-10-25 02:34:39 +00:00
Author
Collaborator
Copy link

[loop:torbscript] claim implement by session:loop-torbscript-20261025-0258 until 2026-10-25T08:30:00Z (CI fix round 1 on PR 479: bootstrap/pack the seed/gates red on e380a73f)

[loop:torbscript] claim implement by session:loop-torbscript-20261025-0258 until 2026-10-25T08:30:00Z (CI fix round 1 on PR 479: bootstrap/pack the seed/gates red on e380a73f)
orion commented 2026-10-25 02:35:44 +00:00
Author
Collaborator
Copy link

[loop:torbscript] release implement: bootstrap failed on 4 parse errors of the seed (juxtaposed calls otherPaths.contains normalized in an if condition, source-tree.trb:140, and content: Some bytes, literal.trb:556); the 611 "captured, not Encode" errors in the tests are follow-on noise of those. Fixed with parentheses, 5e7874f7, next: waiting (CI on PR 479)

[loop:torbscript] release implement: bootstrap failed on 4 parse errors of the seed (juxtaposed calls `otherPaths.contains normalized` in an `if` condition, source-tree.trb:140, and `content: Some bytes`, literal.trb:556); the 611 "captured, not Encode" errors in the tests are follow-on noise of those. Fixed with parentheses, 5e7874f7, next: waiting (CI on PR 479)
orion commented 2026-10-25 03:48:17 +00:00
Author
Collaborator
Copy link

[loop:torbscript] claim implement by session:loop-torbscript-20261025-0258 until 2026-10-25T09:30:00Z (CI fix round 2 on PR 479: bootstrap (linux-x64) red on 5e7874f7, run 1710 job 4)

[loop:torbscript] claim implement by session:loop-torbscript-20261025-0258 until 2026-10-25T09:30:00Z (CI fix round 2 on PR 479: bootstrap (linux-x64) red on 5e7874f7, run 1710 job 4)
orion commented 2026-10-25 03:49:57 +00:00
Author
Collaborator
Copy link

[loop:torbscript] release implement: CI fix round 2 Workspace held a function field (readResource), which made Workspace and everything holding it (Lowering, Checked, Probed, ...) non-Encode, so the seed rejected 593 assert captures in the existing compiler tests; the field is now private (skipped by the derived Encode) behind Workspace.withResources/resource. 64177f5b, next: waiting (CI on PR 479)

[loop:torbscript] release implement: CI fix round 2 Workspace held a function field (readResource), which made Workspace and everything holding it (Lowering, Checked, Probed, ...) non-Encode, so the seed rejected 593 assert captures in the existing compiler tests; the field is now private (skipped by the derived Encode) behind Workspace.withResources/resource. 64177f5b, next: waiting (CI on PR 479)
orion commented 2026-10-25 04:52:42 +00:00
Author
Collaborator
Copy link

[loop:torbscript] claim implement by session:loop-torbscript-20261025-0542 until 2026-10-25T10:50:00Z (PR 479 is not mergeable: main moved to 44a35916, rebase and resolve conflicts)

[loop:torbscript] claim implement by session:loop-torbscript-20261025-0542 until 2026-10-25T10:50:00Z (PR 479 is not mergeable: main moved to 44a35916, rebase and resolve conflicts)
orion commented 2026-10-25 06:14:26 +00:00
Author
Collaborator
Copy link

[loop:torbscript] release implement: PR 479 rebased onto main 44a35916 (head 447f3497, mergeable again, worker ended without a result comment), next: waiting (CI + review on PR 479)

[loop:torbscript] release implement: PR 479 rebased onto main 44a35916 (head 447f3497, mergeable again, worker ended without a result comment), next: waiting (CI + review on PR 479)
orion commented 2026-10-25 07:12:45 +00:00
Author
Collaborator
Copy link

[loop:torbscript] claim implement by session:loop-torbscript-20261025-0542 until 2026-10-25T13:15:00Z (CI fix round on PR 479: tier A (linux-x64) and gates red on 447f3497, run 1728 jobs 5 and 2)

[loop:torbscript] claim implement by session:loop-torbscript-20261025-0542 until 2026-10-25T13:15:00Z (CI fix round on PR 479: tier A (linux-x64) and gates red on 447f3497, run 1728 jobs 5 and 2)
orion commented 2026-10-25 07:35:53 +00:00
Author
Collaborator
Copy link

[loop:torbscript] implement checkpoint (CI fix round): root cause of the red run on 447f3497 was this PR, not main and not a flake. ci / tier A (linux-x64) failed only its format --check lane: compiler/src/project/workspace.trb, compiler/src/semantics/check.trb and compiler/src/semantics/checker/literal.trb (lines this PR added) were not in the torb format canon (parenthesized calls where the formatter writes commands). ci / gates is the reusable-workflow aggregate and failed only because of that lane; every other lane of tier A and all other jobs were green. Fix: chore(checker): run torb format on the resource embedding sources (995529bd), formatted with the nightly-20261025 torb, tools/check-commits.sh clean, linear history, pushed to loop/135-resource-embedding. A new CI run on 995529bd is queued on the runners and has not finished yet; next: wait for CI result on 995529bd (worktree code/wt/torbscript-language-135 kept). Worker tokens: ~85k.

[loop:torbscript] implement checkpoint (CI fix round): root cause of the red run on 447f3497 was this PR, not main and not a flake. `ci / tier A (linux-x64)` failed only its `format --check` lane: `compiler/src/project/workspace.trb`, `compiler/src/semantics/check.trb` and `compiler/src/semantics/checker/literal.trb` (lines this PR added) were not in the `torb format` canon (parenthesized calls where the formatter writes commands). `ci / gates` is the reusable-workflow aggregate and failed only because of that lane; every other lane of tier A and all other jobs were green. Fix: `chore(checker): run torb format on the resource embedding sources` (995529bd), formatted with the nightly-20261025 `torb`, `tools/check-commits.sh` clean, linear history, pushed to loop/135-resource-embedding. A new CI run on 995529bd is queued on the runners and has not finished yet; next: wait for CI result on 995529bd (worktree code/wt/torbscript-language-135 kept). Worker tokens: ~85k.
orion commented 2026-10-25 07:36:33 +00:00
Author
Collaborator
Copy link

[loop:torbscript] release implement: format fix 995529bd pushed to PR 479 (tier A failed only its format lane), worker ended while CI runs, next: waiting (CI + review on PR 479)

[loop:torbscript] release implement: format fix 995529bd pushed to PR 479 (tier A failed only its format lane), worker ended while CI runs, next: waiting (CI + review on PR 479)
orion commented 2026-10-25 11:22:31 +00:00
Author
Collaborator
Copy link

[loop:torbscript] close: PR 479 landed on main as f630c25e (f630c25e33), main CI green on it (13 success, 11 skipped, 0 failed). Closing.

[loop:torbscript] close: PR 479 landed on main as f630c25e (https://git.torb.dev/torbscript/language/commit/f630c25e3368a338248ec02c65d412b39c5aafaa), main CI green on it (13 success, 11 skipped, 0 failed). Closing.
orion closed this issue 2026-10-25 11:22:32 +00:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
loop/705-default-shadow
loop/691-tensor-core
loop/705-fix-candidate
orion/456-tierb-probe
loop/164-measure2
loop/207-measure
loop/164-prototype
probe/207-count
loop/205-measure
loop/140-hide-staged
nightly-20261103
nightly-20261031
nightly-20261029
nightly-20261027
nightly-20261026
nightly-20261025
nightly-20261024
nightly-20261023
nightly-20261021
nightly-20261020
nightly-20261015
nightly-20261014
nightly-20261013
nightly-20261011
nightly-20261010
nightly-20261009
nightly-20261008
seeds
Labels
Clear labels   
loop
Backlog item of the project loop (Orion), see CONTRIBUTING

  
ready
Refined to the definition of ready

No labels
loop
ready
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
torbscript/language#135
No description provided.