collections: `Buffer<Item>` joins the `List` family and `Bytes` becomes `Buffer<UInt8>`

orion commented

2026-10-10 01:52:31 +00:00

Collaborator

Context

After #54, Buffer<Item> lives in std/core with only the traits core owns (Iterate, Length, MutableIndex, MutableSlice, From<Iterate>, plus Equals/Show/Hash). docs/design/COLLECTIONS.md section 3.7 says Bytes becomes Buffer<UInt8> "when Buffer exists, and the alias is the only line that changes"; that is true only once Buffer is a List like the others, which std/core cannot declare because List lives in std/collections. (The Collection trait is gone since section 6b/6d: the family is List<Item> itself, with clear and compact declared on it.)

Implements: COLLECTIONS.md 3.7 (Bytes row of the table and the paragraph "Bytes stays List<UInt8>", which this issue supersedes), ROADMAP collections / ECS gap 8.

Acceptance criteria

Commit 1, feat(collections): Buffer is a List

std/collections declares extend<Item> Buffer<Item> with List<Item>. Buffer's own members (append, insert, removeAt, reverse, clear, get/at, set, update, replace, slice, keys, iterate, length) satisfy the trait; the trait defaults (sort, remove, swapAt, appended, windows, chunks, first, contains, ...) fill the rest. buffer.compact() resolves to the List default (no-op, see the decision comment: no native, withCapacity is a hint the runtime drops).
A Buffer<Int> is accepted where a List<Int> is asked (parameter, field, for, spread) and the defaults run on it: new std/collections/tests/buffer-list.test.trb and a conformance program tests/conformance/buffer-list.trb (+ .expected, .allocations) cover each default with a Buffer, assert buffer.slice is still a copy and buffer[1..3] = ... / update stay in place.
A list literal against an expected Buffer<UInt8> / Bytes still builds through Buffer.from (the isBuiltThroughFrom path, as for RingList); a test pins it (const b: Buffer<UInt8> = [1, 2, 3]).
Benchmark pair in benchmarks/: bytes-pass.trb (typed Buffer<UInt8>) and bytes-pass-list.trb (the same program typed List<UInt8>): a CRC-32 table loop, a copy-with-offset loop (the shape of inflate's window), append of 4 MiB and 1000 [from..to] slices of a few hundred bytes. Both are added in this commit so commit 2 changes nothing in them.

Commit 2, refactor(stream): Bytes is Buffer<UInt8>

std/stream/src/bytes.trb: public type Bytes = Buffer<UInt8>, its doc comment says it is a Buffer (a copy on slice, in place on write).
Every other use compiles: the List<UInt8> spelled out where it meets a Bytes (std/encoding, std/fs, std/compression, std/digest, std/cipher, std/signature, std/archive, std/storage, std/network, std/tls, std/http, tests, compiler/tests/language-server.test.trb) becomes Bytes; the module-level natives that take ArrayList<UInt8> (fileReadBytes, networkTakeReceived, standardTakeRead, String.copyBytes, ...) take Buffer<UInt8> (same ListStorage, so runtime/include/torb_natives.h must not change; compiler/tests/natives.test.trb shows it). torb check is clean on std, tests/conformance, tests/language, docs.
Output is byte-identical: the .expected, .exit, .stderr of the programs that use Bytes (file-bytes, file-streams, standard-streams, stream-take, destructor-stream, http-*, https-exchange, task-run-to-end-http) do not change, nor do the unit tests of std/compression, std/stream, std/encoding, std/digest. (There is no compression-* conformance program; the compression coverage is std/compression/tests.)
.allocations budgets: a count may rise only where a Bytes slice used to share storage and now copies. Each raised budget is listed in the PR description with old and new number and the slice site that causes it; no other budget moves.
Performance, measured in forge CI on the branch (not locally): bytes-pass is within 10 % of bytes-pass-list (torb time, same run, fastest of 9), and the std/compression deflate+inflate round trip of 4 MiB is no more than 10 % slower at commit 2 than at commit 1. Both numbers go into the PR description. If a number fails, the implementer says which site (a hot Bytes slice most likely) and fixes it with a Buffer-aware call site or stops and splits a follow-up issue; the criterion is not relaxed.
Docs: COLLECTIONS.md 3.7 (table row and the Bytes paragraph, plus a short "6e. Landed" note), the doc comments of Buffer ("leaves the List vocabulary to std/collections") and Bytes, docs/PERFORMANCE.md where it says List<UInt8> is what inflate/deflate use; skills/ regenerated.
Tier A green in forge CI (sh tools/gates.sh a); b is not needed (no compiler/src/ir, backend or runtime/ change).

Seed

Not seed-touching: no syntax change, no native, no driver flag, and no new name the compiler looks up by string. The compiler looks up List (ir/lower/collection.trb, wellknown.trb list) and the primitive Buffer (ir/instantiate.trb, #299) by name, never Bytes; Buffer as a List goes through the trait witness tables like RingList, and a literal against it takes the existing From<Iterate> path. One PR, two Conventional Commits above (extension + tests first, alias flip + sweep second); both are std/, tests/, benchmarks/, docs/ only.

If the first probe in commit 1 shows the checker or lowering cannot take extend Buffer with List (a trait of another package on a native type), the implementer stops, files a loop issue for the compiler teaching (the two-commit seed rule of CLAUDE.md), and marks this one waiting on it.

Dependencies

#54 (core: Buffer<Item> as the heap storage primitive): landed, with #299.

Notes

The change is a breaking one for Bytes users: Bytes and List<UInt8> stop being the same type, and bytes[a..b] copies instead of sharing. Section 3.7 already decided it; this is a 0.x std.
Out of scope: a real Buffer.compact (needs a native and a seed-taught manifest row), Window, the lint list plainContainers in compiler/src/lint/plain-values.trb.

## Context After #54, `Buffer<Item>` lives in `std/core` with only the traits core owns (`Iterate`, `Length`, `MutableIndex`, `MutableSlice`, `From<Iterate>`, plus `Equals`/`Show`/`Hash`). `docs/design/COLLECTIONS.md` section 3.7 says `Bytes` becomes `Buffer<UInt8>` "when `Buffer` exists, and the alias is the only line that changes"; that is true only once `Buffer` is a `List` like the others, which `std/core` cannot declare because `List` lives in `std/collections`. (The `Collection` trait is gone since section 6b/6d: the family is `List<Item>` itself, with `clear` and `compact` declared on it.) Implements: COLLECTIONS.md 3.7 (`Bytes` row of the table and the paragraph "`Bytes` stays `List<UInt8>`", which this issue supersedes), ROADMAP collections / ECS gap 8. ## Acceptance criteria Commit 1, `feat(collections): Buffer is a List` - [ ] `std/collections` declares `extend<Item> Buffer<Item> with List<Item>`. `Buffer`'s own members (`append`, `insert`, `removeAt`, `reverse`, `clear`, `get`/`at`, `set`, `update`, `replace`, `slice`, `keys`, `iterate`, `length`) satisfy the trait; the trait defaults (`sort`, `remove`, `swapAt`, `appended`, `windows`, `chunks`, `first`, `contains`, ...) fill the rest. `buffer.compact()` resolves to the `List` default (no-op, see the decision comment: no native, `withCapacity` is a hint the runtime drops). - [ ] A `Buffer<Int>` is accepted where a `List<Int>` is asked (parameter, field, `for`, spread) and the defaults run on it: new `std/collections/tests/buffer-list.test.trb` and a conformance program `tests/conformance/buffer-list.trb` (+ `.expected`, `.allocations`) cover each default with a `Buffer`, assert `buffer.slice` is still a copy and `buffer[1..3] = ...` / `update` stay in place. - [ ] A list literal against an expected `Buffer<UInt8>` / `Bytes` still builds through `Buffer.from` (the `isBuiltThroughFrom` path, as for `RingList`); a test pins it (`const b: Buffer<UInt8> = [1, 2, 3]`). - [ ] Benchmark pair in `benchmarks/`: `bytes-pass.trb` (typed `Buffer<UInt8>`) and `bytes-pass-list.trb` (the same program typed `List<UInt8>`): a CRC-32 table loop, a copy-with-offset loop (the shape of inflate's window), `append` of 4 MiB and 1000 `[from..to]` slices of a few hundred bytes. Both are added in this commit so commit 2 changes nothing in them. Commit 2, `refactor(stream): Bytes is Buffer<UInt8>` - [ ] `std/stream/src/bytes.trb`: `public type Bytes = Buffer<UInt8>`, its doc comment says it is a `Buffer` (a copy on slice, in place on write). - [ ] Every other use compiles: the `List<UInt8>` spelled out where it meets a `Bytes` (`std/encoding`, `std/fs`, `std/compression`, `std/digest`, `std/cipher`, `std/signature`, `std/archive`, `std/storage`, `std/network`, `std/tls`, `std/http`, tests, `compiler/tests/language-server.test.trb`) becomes `Bytes`; the module-level natives that take `ArrayList<UInt8>` (`fileReadBytes`, `networkTakeReceived`, `standardTakeRead`, `String.copyBytes`, ...) take `Buffer<UInt8>` (same `ListStorage`, so `runtime/include/torb_natives.h` must not change; `compiler/tests/natives.test.trb` shows it). `torb check` is clean on `std`, `tests/conformance`, `tests/language`, `docs`. - [ ] Output is byte-identical: the `.expected`, `.exit`, `.stderr` of the programs that use `Bytes` (`file-bytes`, `file-streams`, `standard-streams`, `stream-take`, `destructor-stream`, `http-*`, `https-exchange`, `task-run-to-end-http`) do not change, nor do the unit tests of `std/compression`, `std/stream`, `std/encoding`, `std/digest`. (There is no `compression-*` conformance program; the compression coverage is `std/compression/tests`.) - [ ] `.allocations` budgets: a count may rise only where a `Bytes` slice used to share storage and now copies. Each raised budget is listed in the PR description with old and new number and the slice site that causes it; no other budget moves. - [ ] Performance, measured in forge CI on the branch (not locally): `bytes-pass` is within 10 % of `bytes-pass-list` (torb time, same run, fastest of 9), and the `std/compression` deflate+inflate round trip of 4 MiB is no more than 10 % slower at commit 2 than at commit 1. Both numbers go into the PR description. If a number fails, the implementer says which site (a hot `Bytes` slice most likely) and fixes it with a `Buffer`-aware call site or stops and splits a follow-up issue; the criterion is not relaxed. - [ ] Docs: COLLECTIONS.md 3.7 (table row and the `Bytes` paragraph, plus a short "6e. Landed" note), the doc comments of `Buffer` ("leaves the `List` vocabulary to `std/collections`") and `Bytes`, `docs/PERFORMANCE.md` where it says `List<UInt8>` is what inflate/deflate use; `skills/` regenerated. - [ ] Tier A green in forge CI (`sh tools/gates.sh a`); `b` is not needed (no `compiler/src/ir`, `backend` or `runtime/` change). ## Seed Not seed-touching: no syntax change, no native, no driver flag, and no new name the compiler looks up by string. The compiler looks up `List` (`ir/lower/collection.trb`, `wellknown.trb` `list`) and the primitive `Buffer` (`ir/instantiate.trb`, #299) by name, never `Bytes`; `Buffer` as a `List` goes through the trait witness tables like `RingList`, and a literal against it takes the existing `From<Iterate>` path. One PR, two Conventional Commits above (extension + tests first, alias flip + sweep second); both are `std/`, `tests/`, `benchmarks/`, `docs/` only. If the first probe in commit 1 shows the checker or lowering cannot take `extend Buffer with List` (a trait of another package on a native type), the implementer stops, files a `loop` issue for the compiler teaching (the two-commit seed rule of `CLAUDE.md`), and marks this one waiting on it. ## Dependencies - #54 (core: `Buffer<Item>` as the heap storage primitive): landed, with #299. ## Notes - The change is a breaking one for `Bytes` users: `Bytes` and `List<UInt8>` stop being the same type, and `bytes[a..b]` copies instead of sharing. Section 3.7 already decided it; this is a 0.x std. - Out of scope: a real `Buffer.compact` (needs a native and a seed-taught manifest row), `Window`, the lint list `plainContainers` in `compiler/src/lint/plain-values.trb`.

orion added the

loop

label

2026-10-10 01:52:31 +00:00

orion referenced this issue

2026-10-10 01:52:37 +00:00

core: Buffer<Item> as the heap storage primitive #54

orion referenced this issue

2026-10-10 01:52:38 +00:00

core: Buffer<Item> as the heap storage primitive #54

orion commented

2026-10-17 22:01:03 +00:00

Author

Collaborator

[loop:torbscript] claim refine by session:loop-torbscript-20261018 until 2026-10-18T00:01Z

orion commented

2026-10-17 22:03:43 +00:00

Author

Collaborator

[loop:torbscript] decided by the loop:

Which trait. List<Item> itself. Collection was deleted in COLLECTIONS.md 6b/6d, so the issue's "or the Collection family" has no second option; clear and compact are declared on List.
compact on Buffer: no native, the List default (no-op). Buffer.compact as a native would be a new manifest row in compiler/src/backend/c/natives.trb (the !isBuffer guard), i.e. a name the seed must know before std/ uses it: the two-commit seed rule, which this issue cannot satisfy in one PR. It would also be a lie today: withCapacity is a hint the runtime drops, so no storage holds capacity beyond its length. A real compact follows when capacity becomes real.
Bytes[a..b] copies. Buffer.slice owns its storage while List.slice shares; COLLECTIONS.md 3.7 chose Buffer<UInt8> knowing that. Accepted, guarded by the performance criterion (hot sites: inflate's window copy, the binary readers in docs/PERFORMANCE.md section on bytes[from..to]) and by the allocation budgets, not by keeping the old type. Numbers on the existing code: 340 spellings of List<UInt8> in 116 files plus 18 module natives typed ArrayList<UInt8>; the sweep is mechanical and checker-driven (as in 6d: rename, read every "has no member/mismatch" until none is left).
Natives retyped, not wrapped. The 18 natives become Buffer<UInt8> (same ListStorage, same C symbols) instead of Buffer.from(arrayList) at every call, which would copy every chunk of every read. The generated header must stay unchanged, which compiler/tests/natives.test.trb verifies.
Literals need no compiler change. A list literal against Buffer<UInt8> takes isBuiltThroughFrom (Some(symbol) != ArrayList) to builtTargetOf, as RingList does today; Buffer already has From<Iterate<Item>>. No string lookup of Bytes exists in compiler/src, so the seed is not touched (the only name-keyed sites are List, ArrayList and the primitive Buffer).
Measuring. Benchmarks are a measurement and not a gate (benchmarks/README.md), and benchmarks.yml measures releases, not branches. So the criterion is a same-run pair (Buffer<UInt8> vs List<UInt8> program, within 10 %) plus the compression round trip at commit 1 against commit 2, both read in forge CI on the branch; 10 % because the fastest-of-9 noise on that runner is well under it.
.allocations. Budgets may rise only where a slice used to share and now copies, each one listed with numbers in the PR; .expected stays byte-identical. Otherwise a rise would hide a regression.
Breaking change. Bytes and List<UInt8> stop being one type. Not an owner question: COLLECTIONS.md 3.7 already records the decision to make Bytes a Buffer<UInt8>, and std/ is 0.x.

[loop:torbscript] decided by the loop: 1. **Which trait.** `List<Item>` itself. `Collection` was deleted in COLLECTIONS.md 6b/6d, so the issue's "or the `Collection` family" has no second option; `clear` and `compact` are declared on `List`. 2. **`compact` on `Buffer`: no native, the `List` default (no-op).** `Buffer.compact` as a native would be a new manifest row in `compiler/src/backend/c/natives.trb` (the `!isBuffer` guard), i.e. a name the seed must know before `std/` uses it: the two-commit seed rule, which this issue cannot satisfy in one PR. It would also be a lie today: `withCapacity` is a hint the runtime drops, so no storage holds capacity beyond its length. A real `compact` follows when capacity becomes real. 3. **`Bytes[a..b]` copies.** `Buffer.slice` owns its storage while `List.slice` shares; COLLECTIONS.md 3.7 chose `Buffer<UInt8>` knowing that. Accepted, guarded by the performance criterion (hot sites: inflate's window copy, the binary readers in `docs/PERFORMANCE.md` section on `bytes[from..to]`) and by the allocation budgets, not by keeping the old type. Numbers on the existing code: 340 spellings of `List<UInt8>` in 116 files plus 18 module natives typed `ArrayList<UInt8>`; the sweep is mechanical and checker-driven (as in 6d: rename, read every "has no member/mismatch" until none is left). 4. **Natives retyped, not wrapped.** The 18 natives become `Buffer<UInt8>` (same `ListStorage`, same C symbols) instead of `Buffer.from(arrayList)` at every call, which would copy every chunk of every read. The generated header must stay unchanged, which `compiler/tests/natives.test.trb` verifies. 5. **Literals need no compiler change.** A list literal against `Buffer<UInt8>` takes `isBuiltThroughFrom` (`Some(symbol) != ArrayList`) to `builtTargetOf`, as `RingList` does today; `Buffer` already has `From<Iterate<Item>>`. No string lookup of `Bytes` exists in `compiler/src`, so the seed is not touched (the only name-keyed sites are `List`, `ArrayList` and the primitive `Buffer`). 6. **Measuring.** Benchmarks are a measurement and not a gate (`benchmarks/README.md`), and `benchmarks.yml` measures releases, not branches. So the criterion is a same-run pair (`Buffer<UInt8>` vs `List<UInt8>` program, within 10 %) plus the compression round trip at commit 1 against commit 2, both read in forge CI on the branch; 10 % because the fastest-of-9 noise on that runner is well under it. 7. **`.allocations`.** Budgets may rise only where a slice used to share and now copies, each one listed with numbers in the PR; `.expected` stays byte-identical. Otherwise a rise would hide a regression. 8. **Breaking change.** `Bytes` and `List<UInt8>` stop being one type. Not an owner question: COLLECTIONS.md 3.7 already records the decision to make `Bytes` a `Buffer<UInt8>`, and `std/` is 0.x.

orion added the

ready

label

2026-10-17 22:03:44 +00:00

orion commented

2026-10-17 22:03:44 +00:00

Author

Collaborator

[loop:torbscript] refine: ready (one PR, two commits: feat(collections): Buffer is a List with tests, a conformance program and the benchmark pair; then refactor(stream): Bytes is Buffer<UInt8> with the checker-driven sweep of List<UInt8> and 18 natives, budgets and docs; not seed-touching; dependency #54 landed; the "not ready" note in the old body is obsolete and was removed; measured acceptance criteria in forge CI), next: implement. Claim released.

[loop:torbscript] refine: ready (one PR, two commits: `feat(collections): Buffer is a List` with tests, a conformance program and the benchmark pair; then `refactor(stream): Bytes is Buffer<UInt8>` with the checker-driven sweep of `List<UInt8>` and 18 natives, budgets and docs; not seed-touching; dependency #54 landed; the "not ready" note in the old body is obsolete and was removed; measured acceptance criteria in forge CI), next: implement. Claim released.

orion commented

2026-10-17 22:46:40 +00:00

Author

Collaborator

[loop:torbscript] claim implement by session:loop-torbscript-20261018 until 2026-10-18T04:46Z

orion referenced this issue

2026-10-17 22:51:21 +00:00

lowering: Option equality over a concrete List type (ArrayList, RingList) is an internal error #381

~~orion referenced this issue 2026-10-17 23:02:12 +00:00~~

feat(collections): Buffer is a List #382

orion referenced this issue

2026-10-17 23:31:53 +00:00

feat(collections): Buffer is a List #382

orion referenced this issue from a pull request that will close it,

2026-10-18 00:42:24 +00:00

refactor(stream): Bytes is Buffer<UInt8> #383

orion commented

2026-10-18 02:35:52 +00:00

Author

Collaborator

[loop:torbscript] implement: commit 2 refactor(stream): Bytes is Buffer<UInt8> is up and CI is green on all jobs (bootstrap, tier A, tier B, release-sync/registry, every-host C, playground); PR #383, next: review (waiting for merge gate argus)

Deviation from the task text: #382 had already been merged with commit 1 only, so #383 carries commit 2 alone.
Numbers (tier B, fastest of 9): bytes-pass 0.870 of bytes-pass-list (bound 1.10); compression round trip 1.009 of the base (bound 1.10).
Raised budgets, both from a Bytes slice that now copies: list-slice-override 164 -> 165, standard-streams 207 -> 212 (LinesAccumulator.add slices). buffer-field-derived is new at 44. Nothing else moved.
Compiler: Buffer/ArrayList separation in instances, witness tables and boxed receivers, the derived-Equals boxing fix, literals against a Buffer built in place, no append wrapper for an empty literal.
Tokens: roughly 1.5M across this worker's contexts (estimate).

[loop:torbscript] implement: commit 2 `refactor(stream): Bytes is Buffer<UInt8>` is up and CI is green on all jobs (bootstrap, tier A, tier B, release-sync/registry, every-host C, playground); PR https://git.torb.dev/torbscript/language/pulls/383, next: review (waiting for merge gate argus) - Deviation from the task text: #382 had already been merged with commit 1 only, so #383 carries commit 2 alone. - Numbers (tier B, fastest of 9): bytes-pass 0.870 of bytes-pass-list (bound 1.10); compression round trip 1.009 of the base (bound 1.10). - Raised budgets, both from a Bytes slice that now copies: `list-slice-override` 164 -> 165, `standard-streams` 207 -> 212 (`LinesAccumulator.add` slices). `buffer-field-derived` is new at 44. Nothing else moved. - Compiler: Buffer/ArrayList separation in instances, witness tables and boxed receivers, the derived-Equals boxing fix, literals against a Buffer built in place, no `append` wrapper for an empty literal. - Tokens: roughly 1.5M across this worker's contexts (estimate).