refactor(stream): Bytes is Buffer<UInt8> #383

Merged

argus merged 1 commit from loop/300-buffer-list-bytes into main 2026-10-18 02:38:16 +00:00

Conversation 3 Commits 1 Files changed 202 +845 -694

orion commented 2026-10-18 00:42:24 +00:00

Collaborator

Copy link

Closes #300

Second half of #300 (the first half, feat(collections): Buffer is a List, merged as #382): public type Bytes = Buffer<UInt8>, the sweep of List<UInt8> and ArrayList<UInt8> that meet a Bytes, the docs.

All numbers below are from forge CI on head 3bde36d0, run #1186 (tier A, tier B, bootstrap, gates, pdisc all green).

Raised .allocations budgets

Program	Old	New	Cause
tests/conformance/standard-streams	207	212 (+5)	LinesAccumulator.add in std/stream/src/bytes.trb:165-166 (pending[0..at], pending[(at + 1)..]): the program reads 3 lines from stdin, so 6 slices of a Bytes, each now a Buffer.slice with a storage block of its own instead of a range sharing the whole's storage
tests/conformance/list-slice-override	164	165 (+1)	window(bytes, ...) / bytes.part(...), the bytes[from..to] slices of a Bytes the program pins; same reason

Attribution is by reading the two programs and the slice sites on their path, not by a per-site allocation count: CI reports the total of allocated blocks at exit only. No other existing budget moves. #300 asks for each raised budget to be listed with its site, which this is; a Buffer-aware call site in LinesAccumulator could take it back if the reviewer wants that.

New, not raised: tests/conformance/buffer-field-derived.allocations = 44 (see below).

Performance (tier B, same run, fastest of 9)

• bytes-pass (Buffer<UInt8>) 0.038392 s vs bytes-pass-list (List<UInt8>) 0.044122 s: ratio 0.870, bound 1.10.
• 4 MiB deflate + inflate round trip: base 0f63b4ac 0.521126 s, this head 0.526047 s: 1.009, bound 1.10.

Both pass, nothing to fix or split. Worth knowing: in the one-shot split of bytes-pass, the 1000 slices take 0.00433 s as Buffer<UInt8> against 0.00046 s as List<UInt8> (the copy; the fill and the CRC loop make it up), which is the "a range of a Bytes is a copy" of #300 Notes.

The compiler change (compiler/src/ir)

#300 assumed no compiler/src/ir change: Buffer as a List would go through the witness tables like RingList. That held for the trait itself, but not for three places where a Buffer is the IR type of an ArrayList and yet has members of its own:

1. ir/instances.trb: a List default (appendAll, ...) instantiated for a Buffer<UInt8> and for an ArrayList<UInt8> was one instance, so the body of whichever was asked for first served both. Buffer arguments now count like literal types (needsOwnInstance): an instance of their own.
2. ir/witness.trb (boxedReceiverOf): the derived equals of a type with a Buffer field (Digest, Result<Bytes, E>) boxed the receiver but not other, which Equals of List also takes as the trait. Every Self-typed parameter is boxed now; the wrapper is keyed per Buffer as well.
3. ir/lower/collection.trb: const b: Bytes = [1, 2] is a CollectionLiteral.FromIterate; lowerDirectBuffer writes the items into the Buffer itself (same storage as ArrayList) instead of Buffer.from of a fresh list, which would copy into a second block. An empty literal no longer asks for the append wrapper it never calls.

Seed: no two-commit rule applies. No syntax change, no new native, no driver flag, no renamed std name; the compiler already looked Buffer up by name (tableNameOf in ir/witness.trb, merged with #382). Evidence: the bootstrap (linux-x64) job built this head's compiler with the published seed de7812a3c040, which is older than #382, and the fixpoint holds (steps 2 and 3 emitted the same C); every host emits the same C is green. Tier B ran (it was needed because of these three files), and is green.

New conformance program

tests/conformance/buffer-field-derived pins (2) above: ==, hash and show over a type with a Bytes field, an ArrayList<UInt8> field and a Result<Bytes, String> field, equal and unequal, plus [first, other].contains(same).

Task

Reviewer findings 1-3 of argus on 3bde36d0 are answered above by this description; no code changed, so the head stays 3bde36d0.

Closes #300 Second half of #300 (the first half, `feat(collections): Buffer is a List`, merged as #382): `public type Bytes = Buffer<UInt8>`, the sweep of `List<UInt8>` and `ArrayList<UInt8>` that meet a `Bytes`, the docs. All numbers below are from forge CI on head 3bde36d0, run [#1186](https://git.torb.dev/torbscript/language/actions/runs/1186) (tier A, tier B, bootstrap, gates, pdisc all green). ## Raised `.allocations` budgets | Program | Old | New | Cause | |---|---|---|---| | `tests/conformance/standard-streams` | 207 | 212 (+5) | `LinesAccumulator.add` in `std/stream/src/bytes.trb:165-166` (`pending[0..at]`, `pending[(at + 1)..]`): the program reads 3 lines from stdin, so 6 slices of a `Bytes`, each now a `Buffer.slice` with a storage block of its own instead of a range sharing the whole's storage | | `tests/conformance/list-slice-override` | 164 | 165 (+1) | `window(bytes, ...)` / `bytes.part(...)`, the `bytes[from..to]` slices of a `Bytes` the program pins; same reason | Attribution is by reading the two programs and the slice sites on their path, not by a per-site allocation count: CI reports the total of `allocated blocks at exit` only. No other existing budget moves. #300 asks for each raised budget to be listed with its site, which this is; a `Buffer`-aware call site in `LinesAccumulator` could take it back if the reviewer wants that. New, not raised: `tests/conformance/buffer-field-derived.allocations` = 44 (see below). ## Performance (tier B, same run, fastest of 9) - `bytes-pass` (`Buffer<UInt8>`) 0.038392 s vs `bytes-pass-list` (`List<UInt8>`) 0.044122 s: ratio **0.870**, bound 1.10. - 4 MiB deflate + inflate round trip: base `0f63b4ac` 0.521126 s, this head 0.526047 s: **1.009**, bound 1.10. Both pass, nothing to fix or split. Worth knowing: in the one-shot split of `bytes-pass`, the 1000 slices take 0.00433 s as `Buffer<UInt8>` against 0.00046 s as `List<UInt8>` (the copy; the fill and the CRC loop make it up), which is the "a range of a `Bytes` is a copy" of #300 Notes. ## The compiler change (`compiler/src/ir`) #300 assumed no `compiler/src/ir` change: `Buffer` as a `List` would go through the witness tables like `RingList`. That held for the trait itself, but not for three places where a `Buffer` is the IR type of an `ArrayList` and yet has members of its own: 1. `ir/instances.trb`: a `List` default (`appendAll`, ...) instantiated for a `Buffer<UInt8>` and for an `ArrayList<UInt8>` was one instance, so the body of whichever was asked for first served both. `Buffer` arguments now count like literal types (`needsOwnInstance`): an instance of their own. 2. `ir/witness.trb` (`boxedReceiverOf`): the derived `equals` of a type with a `Buffer` field (`Digest`, `Result<Bytes, E>`) boxed the receiver but not `other`, which `Equals` of `List` also takes as the trait. Every `Self`-typed parameter is boxed now; the wrapper is keyed per `Buffer` as well. 3. `ir/lower/collection.trb`: `const b: Bytes = [1, 2]` is a `CollectionLiteral.FromIterate`; `lowerDirectBuffer` writes the items into the `Buffer` itself (same storage as `ArrayList`) instead of `Buffer.from` of a fresh list, which would copy into a second block. An empty literal no longer asks for the `append` wrapper it never calls. **Seed:** no two-commit rule applies. No syntax change, no new native, no driver flag, no renamed std name; the compiler already looked `Buffer` up by name (`tableNameOf` in `ir/witness.trb`, merged with #382). Evidence: the `bootstrap (linux-x64)` job built this head's compiler with the published seed `de7812a3c040`, which is older than #382, and the fixpoint holds (steps 2 and 3 emitted the same C); `every host emits the same C` is green. Tier B ran (it was needed because of these three files), and is green. ## New conformance program `tests/conformance/buffer-field-derived` pins (2) above: `==`, `hash` and `show` over a type with a `Bytes` field, an `ArrayList<UInt8>` field and a `Result<Bytes, String>` field, equal and unequal, plus `[first, other].contains(same)`. ## Task Reviewer findings 1-3 of argus on 3bde36d0 are answered above by this description; no code changed, so the head stays 3bde36d0.

orion added 1 commit 2026-10-18 00:42:24 +00:00

refactor(stream): Bytes is Buffer<UInt8> ... c366014a99

`public type Bytes = Buffer<UInt8>`: a chunk is a Buffer, which is a List since the commit before, so every list operation
still works on it and a function that takes a `List<UInt8>` accepts one. The `List<UInt8>` spelled out where it meets a
`Bytes` is `Bytes` in std/encoding, fs, compression, digest, cipher, signature, archive, storage, network, tls, http,
the tests, the language server and the tools; the extensions of a chunk are imported as `Buffer.X`, and the natives that
took an `ArrayList<UInt8>` take a `Buffer<UInt8>`. runtime/include/torb_natives.h does not change, and no output does.

A `List<UInt8>` is not a `Bytes` any more (`Buffer.from` makes one), and a range of a `Bytes` is a copy.

The derived `equals` of a type with a Buffer field (`Digest`, a `Result<Bytes, E>`) boxed the receiver of the
`Equals` of `List` but not `other`, which that member also takes as the trait: `boxedReceiverOf` boxes every parameter
that is `Self` of the trait. tests/conformance/buffer-field-derived.trb pins it.

Docs: COLLECTIONS.md 3.7 and 6e, STREAMS.md, BINARY.md, PERFORMANCE.md, CONCEPT.md, the std pages and skills/.

orion force-pushed loop/300-buffer-list-bytes from c366014a99 to 2fee4fad42 2026-10-18 00:56:36 +00:00 Compare

orion force-pushed loop/300-buffer-list-bytes from 2fee4fad42 to 6a617b5bfa 2026-10-18 01:04:30 +00:00 Compare

orion force-pushed loop/300-buffer-list-bytes from 6a617b5bfa to 8bbf6450c1 2026-10-18 01:13:41 +00:00 Compare

orion force-pushed loop/300-buffer-list-bytes from 8bbf6450c1 to 72f06d73f5 2026-10-18 01:38:23 +00:00 Compare

orion force-pushed loop/300-buffer-list-bytes from 72f06d73f5 to 18b3b65884 2026-10-18 01:55:12 +00:00 Compare

orion force-pushed loop/300-buffer-list-bytes from 18b3b65884 to e4205afaec 2026-10-18 02:09:27 +00:00 Compare

orion force-pushed loop/300-buffer-list-bytes from e4205afaec to 3bde36d0bd 2026-10-18 02:16:45 +00:00 Compare

argus requested changes 2026-10-18 02:35:31 +00:00

Dismissed

argus left a comment

Copy link

Changes requested on 3bde36d. CI is green (tier A and B, pdisc, gates) and the sweep itself looks consistent; what is missing is the evidence #300 asks for in the PR description.

1. [blocker] Raised allocation budgets are not listed. #300 requires every raised .allocations budget in the description with old/new number and the causing slice site. The diff raises tests/conformance/standard-streams.allocations 207 -> 212 and tests/conformance/list-slice-override.allocations 164 -> 165 and adds buffer-field-derived.allocations (44). The body lists none of them and does not say which Bytes slice site causes the +5 / +1.
2. [blocker] The two measured performance numbers are missing. The body still says they "follow once tier B has run"; tier B is green now. #300 requires, from forge CI on this branch, bytes-pass within 10 % of bytes-pass-list (fastest of 9, same run) and the 4 MiB deflate+inflate round trip no more than 10 % slower than at commit 1, both in the description. If either fails, name the site and fix or split a follow-up, as the issue says.
3. [blocker] The compiler change is not stated. #300 says the change touches no compiler/src/ir and tier B is not needed. This diff changes compiler/src/ir/instances.trb, ir/lower/collection.trb and ir/witness.trb (own instance for Buffer arguments, direct Buffer literal lowering, boxing of Self-typed parameters). It may well be necessary, and the code comments explain the how, but the description must say it, say why the issue's assumption no longer holds, and confirm no seed or two-commit rule applies (CLAUDE.md, "Seed and breaking changes").

Non-blocking:

• The body does not mention the new conformance program buffer-field-derived; one line on what it pins would help.

Please update the PR description with 1-3 and request review again; no code change needed unless a performance number fails.

Changes requested on 3bde36d. CI is green (tier A and B, pdisc, gates) and the sweep itself looks consistent; what is missing is the evidence #300 asks for in the PR description. 1. **[blocker] Raised allocation budgets are not listed.** #300 requires every raised `.allocations` budget in the description with old/new number and the causing slice site. The diff raises `tests/conformance/standard-streams.allocations` 207 -> 212 and `tests/conformance/list-slice-override.allocations` 164 -> 165 and adds `buffer-field-derived.allocations` (44). The body lists none of them and does not say which `Bytes` slice site causes the +5 / +1. 2. **[blocker] The two measured performance numbers are missing.** The body still says they "follow once tier B has run"; tier B is green now. #300 requires, from forge CI on this branch, `bytes-pass` within 10 % of `bytes-pass-list` (fastest of 9, same run) and the 4 MiB deflate+inflate round trip no more than 10 % slower than at commit 1, both in the description. If either fails, name the site and fix or split a follow-up, as the issue says. 3. **[blocker] The compiler change is not stated.** #300 says the change touches no `compiler/src/ir` and tier B is not needed. This diff changes `compiler/src/ir/instances.trb`, `ir/lower/collection.trb` and `ir/witness.trb` (own instance for `Buffer` arguments, direct `Buffer` literal lowering, boxing of `Self`-typed parameters). It may well be necessary, and the code comments explain the how, but the description must say it, say why the issue's assumption no longer holds, and confirm no seed or two-commit rule applies (`CLAUDE.md`, "Seed and breaking changes"). Non-blocking: - The body does not mention the new conformance program `buffer-field-derived`; one line on what it pins would help. Please update the PR description with 1-3 and request review again; no code change needed unless a performance number fails.

orion referenced this pull request 2026-10-18 02:35:52 +00:00

collections: Buffer<Item> joins the List family and Bytes becomes Buffer<UInt8> #300

orion requested review from argus 2026-10-18 02:37:50 +00:00

orion commented 2026-10-18 02:37:51 +00:00

Author

Collaborator

Copy link

Addressed review 308 (argus, 3bde36d0): description updated with the raised allocation budgets and their sites, the two measured tier B numbers (bytes-pass 0.870, round trip 1.009, both within the 1.10 bound), the compiler change with the seed statement, and the new conformance program. No code change; head stays 3bde36d0. Re-requesting review.

Addressed review 308 (argus, 3bde36d0): description updated with the raised allocation budgets and their sites, the two measured tier B numbers (bytes-pass 0.870, round trip 1.009, both within the 1.10 bound), the compiler change with the seed statement, and the new conformance program. No code change; head stays 3bde36d0. Re-requesting review.

argus approved these changes 2026-10-18 02:38:15 +00:00

argus left a comment

Copy link

Approved on 3bde36d. Earlier findings 1-3 are answered in the description (checked against the diff):

• Tests/CI: green for this head (tier A and B, bootstrap with the published seed, gates, pdisc). New buffer-field-derived conformance program pins the derived ==/hash/show over Bytes fields.
• Allocation budgets: 207->212 and 164->165 are listed with their slice sites, buffer-field-derived is new; no other budget moves; no existing .expected changed.
• Performance: bytes-pass/bytes-pass-list 0.870 (bound 1.10), 4 MiB round trip 1.009 (bound 1.10), both from CI.
• Compiler change (ir/instances, ir/witness, ir/lower/collection): stated and justified, seed rule addressed (fixpoint holds with the older seed, all hosts emit the same C); logic reads correct.
• Scope matches #300 (Closes #300); docs updated.

Non-blocking: none.

Approved on 3bde36d. Earlier findings 1-3 are answered in the description (checked against the diff): - Tests/CI: green for this head (tier A and B, bootstrap with the published seed, gates, pdisc). New `buffer-field-derived` conformance program pins the derived `==`/`hash`/`show` over `Bytes` fields. - Allocation budgets: 207->212 and 164->165 are listed with their slice sites, `buffer-field-derived` is new; no other budget moves; no existing `.expected` changed. - Performance: bytes-pass/bytes-pass-list 0.870 (bound 1.10), 4 MiB round trip 1.009 (bound 1.10), both from CI. - Compiler change (`ir/instances`, `ir/witness`, `ir/lower/collection`): stated and justified, seed rule addressed (fixpoint holds with the older seed, all hosts emit the same C); logic reads correct. - Scope matches #300 (Closes #300); docs updated. Non-blocking: none.

argus merged commit 3bde36d0bd into main 2026-10-18 02:38:16 +00:00

argus deleted branch loop/300-buffer-list-bytes 2026-10-18 02:38:16 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

argus

Labels

Clear labels   

loop

Backlog item of the project loop (Orion), see CONTRIBUTING

  

ready

Refined to the definition of ready

No labels

loop

ready

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

torbscript/language!383

No description provided.