js: lists, buffers and text natives under Node (slice 4A of #63) #708

New issue

Open

opened 2026-10-30 22:28:41 +00:00 by orion · 0 comments

orion commented

2026-10-30 22:28:41 +00:00

Collaborator

Context

Step 5 of epic #63 (JavaScript back end), slice 4, part A of docs/design/JAVASCRIPT-BACKEND.md section 6 row 4 ("The kernel, part 1"). Slices 1 to 3 (#678, #684, #701) are on main: compiler/src/backend/javascript/{emit,body,type,natives,witness,count,writer}.trb, runtime/javascript/{core,int64}.mjs, tools/conformance.sh --js, tests/conformance/js.allow, CI job conformance-js. The integer, comparison, conversion and text intrinsics (K1, K2, K4, K7, TextConcat, TextEqual, TextByteLength) are already emitted by emitIntrinsic; what stops every program that holds a list is the runtime natives (NativeTarget.Runtime, nativeNameOf in natives.trb knows only the five torb_show_* / torb_process_exit symbols) and Instruction.ContainerNew(kind: ListStorage) (body.trb reports it as unsupported).

Slice 4 of the record is too big for one PR (kernel natives, ListStorage, the manifest walk, BigInt property tests, isSame). Decided by the loop: it is split in two. This issue is 4A, the storage and the natives the standard library's list code stands on. 4B (filed when 4A lands): the walk of nativeEntries() that fails for an entry without JavaScript (with the explicit deferred list for slices 5 and 7), isSame, the Intrinsic sweep over every width, and the property tests against BigInt in tools/.

Source

docs/design/JAVASCRIPT-BACKEND.md section 3 (K5 row), section 4 ("Collections": ListStorage is a growing Array or typed array with a count) and section 6 row 4.
docs/design/NATIVES.md "Buffer", "String" and the list entries (listEntries in compiler/src/backend/c/natives.trb: length, get, append, set, insert, removeAt, replaceBetween, reverse, clear, compact, sliceBetween, withCapacity) and the String kernel rows sliceBytes, byteAt.
The C runtime is the reference for what each symbol means: runtime/ (torb_list_*), and compiler/src/backend/c/body.trb for ContainerNew and for how NativeResult.Optional, ownedParameters and addressedParameters shape a call.

Goal

Programs that build, index, append to, slice and print lists run under Node:

ListStorage. ContainerNew(ListStorage) creates a list object (a growing Array; a plain Array for every element type in this slice, typed arrays are an optimisation for a later change). Every symbol listEntries names is a function of runtime/javascript/ with the C symbol's name and meaning: bounds errors panic with the same message and location as torb_list_* (torb_location is the place string emitIntrinsic already passes); an Optional result (get, removeAt) returns the value or undefined according to how emitCall shapes an Option today (read the existing Option construct in body.trb, do not invent a second form).
Elements. The element copy rule of slice 2 holds: an element that is an aggregate is cloned by the generated clone() when it is read out (get), stored in (append, set, insert) or moved into a slice; a list of shared type objects shares the objects and retains them, and Release of a list closes held Close objects in the order the C torb_list_* drop does (slice 3's release$T extended by a list case, only when Closing.holding says the list can hold one).
Text. String.sliceBytes and String.byteAt (K5), over the representation of section 2.5 as slices 1 and 2 left it; String.add is already +.
Natives table. natives.trb gains the symbols above; a symbol the back end still lacks stays a finding naming the symbol, as today.
Everything a program does not use stays absent from the output: a program with no list adds no list code (pinned by an unchanged output of javascript_first.trb and js-witnesses-close.trb).

Decided by the loop

File layout. The list natives go into a new runtime/javascript/list.mjs, which the driver reads and writes in front of the program only when the emitted program reaches a ListStorage symbol (one whole-program flag from emit.trb, like rc in slice 3); core.mjs is unchanged except for a shared helper if two files need it. Reason: core.mjs is in every program, and the record's rule is that unused machinery is absent.
No MapStorage or SetStorage here. Section 4 waits for epic #55 (collection storage on Buffer) for maps and sets; programs that need a HashMap, HashSet or Trie* stay off js.allow.
Capacity. withCapacity drops its hint as the C runtime does (first version), so no preallocation.
Panics. A bounds or negative-index panic goes through the existing torb_panic path with the at location; exit code and stderr match the C leg.

Files

compiler/src/backend/javascript/body.trb (ContainerNew, the Runtime call shapes for Optional / owned / addressed parameters), natives.trb (symbols), type.trb (list case of the drop / release$T, clone() of a list element), emit.trb (the list flag, reading list.mjs).
runtime/javascript/list.mjs (new).
tests/conformance/js-lists.trb (new) with .expected and .exit, and a second program js-lists-panic.trb with .expected, .stderr, .exit for an out-of-range set/insert (check an existing panic program of tests/conformance for the exact file set before writing it); tests/conformance/js.allow.
docs/design/JAVASCRIPT-BACKEND.md: row 4 split into 4A (done) and 4B, and a "What slice 4A left out" paragraph; docs regenerated as torb docs requires.

Tests

New program js-lists.trb, text and integers only: build a list by append and a literal, index with get/[], set, insert, removeAt, replaceBetween, reverse, clear, sliceBetween (and the slice keeps its own copy), a list of records (aggregate copy: mutating the original after append does not change the stored one), a list of shared type objects with close() (closed in the C drop order when the list is released), iteration with for, print of a list through Show, sliceBytes / byteAt on ASCII and multi-byte text. Same .expected and .exit on the C, VM and JavaScript legs.
Existing programs that build and run under Node with the same .expected, .stderr and .exit files join js.allow. Candidates (list-using programs that need no map, set, float, task, file or time): array-pattern-binding, bankers-queue, buffer-list, buffer-holds-close, buffer-field-derived, collect-through-wrappers, cons-stack-persists, declared-list-shape, destructor-fields-and-lists, destructor-patterns-and-temporaries, destructor-slice-concrete, destructor-slice, heap-order, interpolation, joined, list-bounds-proven, plus the trait programs slice 3 left off (bound-methods, plain-bound, generic-trait-members, trait-constants, source-trait-value, trait-typed-literals, entry-cell-trait, shared-top-level-const, destructor-early-exits, slice-close-generic). A program that still fails for a construct of a later slice (a map or set, a float, a task, a native of 4B) stays off; the PR body lists each with the finding or output line that names the later slice. Programs with .allocations files: the counts are a C-leg check; the printed close() lines must match.
No program of js.allow is removed; the existing JS programs still pass.

Acceptance criteria

js-lists.trb and js-lists-panic.trb pass on the C, VM and JavaScript legs with identical expected files
js.allow grows by every program of the candidate list that passes, at least buffer-list, heap-order and one of destructor-fields-and-lists / destructor-slice; each program left off is listed in the PR with its reason
A program without a list produces the same JavaScript as before (no list.mjs in front of it): pinned by an unchanged output of javascript_first.trb
Row 4 of the design record is split into 4A (done) and 4B; docs regenerated; torb format --check and torb docs check green
tools/check-commits.sh clean; sh tools/gates.sh a and the conformance-js job green in forge CI (CI only, no local bootstrap or gates)

Seed

Compiler and runtime only: no syntax change, no new native, no std name the compiler looks up, no change of the manifest (nativeEntries() is read, not edited), no Architecture case. seed/ is untouched. One PR; one Conventional Commit per logical change (feat(js): ..., test(js): ...), expect two or three. If the list-drop order work grows past one review, land ListStorage and the natives first and move the close() case into a follow-up commit of the same PR (not a second issue).

Out of scope

4B (manifest walk, isSame, Intrinsic sweep, BigInt property tests), MapStorage / SetStorage (epic #55), floats and libm (5), structured control flow (6), tasks (7), .nojs markers and torb test (8), typed arrays for lists, a minifier, source maps.

Dependencies

#701 (slice 3, landed as PR #703).

Part of #63 (epic; do not close it).

## Context Step 5 of epic #63 (JavaScript back end), slice 4, part A of `docs/design/JAVASCRIPT-BACKEND.md` section 6 row 4 ("The kernel, part 1"). Slices 1 to 3 (#678, #684, #701) are on `main`: `compiler/src/backend/javascript/{emit,body,type,natives,witness,count,writer}.trb`, `runtime/javascript/{core,int64}.mjs`, `tools/conformance.sh --js`, `tests/conformance/js.allow`, CI job `conformance-js`. The integer, comparison, conversion and text intrinsics (K1, K2, K4, K7, `TextConcat`, `TextEqual`, `TextByteLength`) are already emitted by `emitIntrinsic`; what stops every program that holds a list is the **runtime natives** (`NativeTarget.Runtime`, `nativeNameOf` in `natives.trb` knows only the five `torb_show_*` / `torb_process_exit` symbols) and `Instruction.ContainerNew(kind: ListStorage)` (`body.trb` reports it as unsupported). Slice 4 of the record is too big for one PR (kernel natives, `ListStorage`, the manifest walk, BigInt property tests, `isSame`). Decided by the loop: it is split in two. **This issue is 4A**, the storage and the natives the standard library's list code stands on. **4B** (filed when 4A lands): the walk of `nativeEntries()` that fails for an entry without JavaScript (with the explicit *deferred* list for slices 5 and 7), `isSame`, the `Intrinsic` sweep over every width, and the property tests against `BigInt` in `tools/`. ## Source - `docs/design/JAVASCRIPT-BACKEND.md` section 3 (K5 row), section 4 ("Collections": `ListStorage` is a growing `Array` or typed array with a count) and section 6 row 4. - `docs/design/NATIVES.md` "Buffer", "String" and the list entries (`listEntries` in `compiler/src/backend/c/natives.trb`: `length`, `get`, `append`, `set`, `insert`, `removeAt`, `replaceBetween`, `reverse`, `clear`, `compact`, `sliceBetween`, `withCapacity`) and the `String` kernel rows `sliceBytes`, `byteAt`. - The C runtime is the reference for what each symbol means: `runtime/` (`torb_list_*`), and `compiler/src/backend/c/body.trb` for `ContainerNew` and for how `NativeResult.Optional`, `ownedParameters` and `addressedParameters` shape a call. ## Goal Programs that build, index, append to, slice and print lists run under Node: - **`ListStorage`.** `ContainerNew(ListStorage)` creates a list object (a growing `Array`; a plain `Array` for every element type in this slice, typed arrays are an optimisation for a later change). Every symbol `listEntries` names is a function of `runtime/javascript/` with the C symbol's name and meaning: bounds errors panic with the **same message and location** as `torb_list_*` (`torb_location` is the `place` string `emitIntrinsic` already passes); an `Optional` result (`get`, `removeAt`) returns the value or `undefined` according to how `emitCall` shapes an `Option` today (read the existing `Option` construct in `body.trb`, do not invent a second form). - **Elements.** The element copy rule of slice 2 holds: an element that is an aggregate is cloned by the generated `clone()` when it is read out (`get`), stored in (`append`, `set`, `insert`) or moved into a slice; a list of `shared type` objects shares the objects and retains them, and `Release` of a list closes held `Close` objects in the order the C `torb_list_*` drop does (slice 3's `release$T` extended by a list case, only when `Closing.holding` says the list can hold one). - **Text.** `String.sliceBytes` and `String.byteAt` (K5), over the representation of section 2.5 as slices 1 and 2 left it; `String.add` is already `+`. - **Natives table.** `natives.trb` gains the symbols above; a symbol the back end still lacks stays a finding naming the symbol, as today. - Everything a program does not use stays absent from the output: a program with no list adds no list code (pinned by an unchanged output of `javascript_first.trb` and `js-witnesses-close.trb`). ## Decided by the loop - **File layout.** The list natives go into a new `runtime/javascript/list.mjs`, which the driver reads and writes in front of the program **only when** the emitted program reaches a `ListStorage` symbol (one whole-program flag from `emit.trb`, like `rc` in slice 3); `core.mjs` is unchanged except for a shared helper if two files need it. Reason: `core.mjs` is in every program, and the record's rule is that unused machinery is absent. - **No `MapStorage` or `SetStorage` here.** Section 4 waits for epic #55 (collection storage on `Buffer`) for maps and sets; programs that need a `HashMap`, `HashSet` or `Trie*` stay off `js.allow`. - **Capacity.** `withCapacity` drops its hint as the C runtime does (first version), so no preallocation. - **Panics.** A bounds or negative-index panic goes through the existing `torb_panic` path with the `at` location; exit code and stderr match the C leg. ## Files - `compiler/src/backend/javascript/body.trb` (`ContainerNew`, the `Runtime` call shapes for `Optional` / owned / addressed parameters), `natives.trb` (symbols), `type.trb` (list case of the drop / `release$T`, `clone()` of a list element), `emit.trb` (the list flag, reading `list.mjs`). - `runtime/javascript/list.mjs` (new). - `tests/conformance/js-lists.trb` (new) with `.expected` and `.exit`, and a second program `js-lists-panic.trb` with `.expected`, `.stderr`, `.exit` for an out-of-range `set`/`insert` (check an existing panic program of `tests/conformance` for the exact file set before writing it); `tests/conformance/js.allow`. - `docs/design/JAVASCRIPT-BACKEND.md`: row 4 split into 4A (done) and 4B, and a "What slice 4A left out" paragraph; docs regenerated as `torb docs` requires. ## Tests 1. **New program `js-lists.trb`**, text and integers only: build a list by `append` and a literal, index with `get`/`[]`, `set`, `insert`, `removeAt`, `replaceBetween`, `reverse`, `clear`, `sliceBetween` (and the slice keeps its own copy), a list of records (aggregate copy: mutating the original after `append` does not change the stored one), a list of `shared type` objects with `close()` (closed in the C drop order when the list is released), iteration with `for`, `print` of a list through `Show`, `sliceBytes` / `byteAt` on ASCII and multi-byte text. Same `.expected` and `.exit` on the C, VM and JavaScript legs. 2. **Existing programs** that build and run under Node with the same `.expected`, `.stderr` and `.exit` files join `js.allow`. Candidates (list-using programs that need no map, set, float, task, file or time): `array-pattern-binding`, `bankers-queue`, `buffer-list`, `buffer-holds-close`, `buffer-field-derived`, `collect-through-wrappers`, `cons-stack-persists`, `declared-list-shape`, `destructor-fields-and-lists`, `destructor-patterns-and-temporaries`, `destructor-slice-concrete`, `destructor-slice`, `heap-order`, `interpolation`, `joined`, `list-bounds-proven`, plus the trait programs slice 3 left off (`bound-methods`, `plain-bound`, `generic-trait-members`, `trait-constants`, `source-trait-value`, `trait-typed-literals`, `entry-cell-trait`, `shared-top-level-const`, `destructor-early-exits`, `slice-close-generic`). A program that still fails for a construct of a later slice (a map or set, a float, a task, a native of 4B) stays off; the PR body lists each with the finding or output line that names the later slice. Programs with `.allocations` files: the counts are a C-leg check; the printed `close()` lines must match. 3. No program of `js.allow` is removed; the existing JS programs still pass. ## Acceptance criteria - [ ] `js-lists.trb` and `js-lists-panic.trb` pass on the C, VM and JavaScript legs with identical expected files - [ ] `js.allow` grows by every program of the candidate list that passes, at least `buffer-list`, `heap-order` and one of `destructor-fields-and-lists` / `destructor-slice`; each program left off is listed in the PR with its reason - [ ] A program without a list produces the same JavaScript as before (no `list.mjs` in front of it): pinned by an unchanged output of `javascript_first.trb` - [ ] Row 4 of the design record is split into 4A (done) and 4B; docs regenerated; `torb format --check` and `torb docs check` green - [ ] `tools/check-commits.sh` clean; `sh tools/gates.sh a` and the `conformance-js` job green in forge CI (CI only, no local bootstrap or gates) ## Seed Compiler and runtime only: no syntax change, no new native, no std name the compiler looks up, no change of the manifest (`nativeEntries()` is read, not edited), no `Architecture` case. `seed/` is untouched. One PR; one Conventional Commit per logical change (`feat(js): ...`, `test(js): ...`), expect two or three. If the list-drop order work grows past one review, land `ListStorage` and the natives first and move the `close()` case into a follow-up commit of the same PR (not a second issue). ## Out of scope 4B (manifest walk, `isSame`, `Intrinsic` sweep, `BigInt` property tests), `MapStorage` / `SetStorage` (epic #55), floats and `libm` (5), structured control flow (6), tasks (7), `.nojs` markers and `torb test` (8), typed arrays for lists, a minifier, source maps. ## Dependencies #701 (slice 3, landed as PR #703). Part of #63 (epic; do not close it).

orion added the

loop

ready

labels

2026-10-30 22:28:41 +00:00

orion referenced this issue

2026-10-30 22:28:50 +00:00

backend: JavaScript #63

orion referenced this issue

2026-10-30 22:28:50 +00:00

backend: JavaScript #63