compression: zstd, snappy and lz4

orion commented

2026-10-05 21:07:22 +00:00

Collaborator

Context

Epic for FIRST-PARTY.md gap 26: zstd, snappy and lz4 in std/compression, which MongoDB's wire compression (snappy, zstd) and Kafka (snappy, lz4, zstd) use; optional for both packages. The design record stays the specification; this epic is the slicing.

Goal

std/compression reads and writes Snappy, LZ4 and Zstandard next to DEFLATE and gzip, as methods of Bytes, in ordinary TorbScript with no native (the precedent of std/compression's DEFLATE and gzip, RELEASE.md section 7.2). Natives for speed are a later, measured decision (benchmark first, then a native under the two-commit rule), not part of the first pass.

Why it is an epic

Three formats with very different weight: the Snappy and LZ4 block formats are a few hundred lines each, the frame formats need new checksums (xxHash32, xxHash64, CRC-32C), and a Zstandard decoder (RFC 8878: frame, literals with Huffman and FSE, sequences, repeat offsets) is far beyond one PR of one or two commits.

Steps (one issue each, filed just in time when it starts next)

1. Snappy and LZ4 block formats: #432
2. xxHash32 and xxHash64 in std/digest, then the LZ4 frame format (Bytes.lz4Framed / lz4Unframed) in std/compression
3. Zstandard decoder: frames, raw/RLE/compressed blocks, Huffman and FSE, content checksum (Bytes.zstdDecompressed)
4. Zstandard encoder: deterministic, greedy, predefined FSE tables, raw and RLE blocks as fallback (Bytes.zstdCompressed)
5. Snappy stream framings (the CRC-32C framing format and the xerial/Kafka block framing); gap 26 marked done

Steps 2 to 5 are listed here so the order is recorded; they become issues only when they start next. Step 3 may be split further (e.g. literals section first) when it is filed.

Design points (decided by the loop, see the comment of 2026-10-23)

Pure TorbScript in std/compression, one file per format; no native, no syntax change, no new driver flag: touches the seed: no (no step needs it; a native added later for performance would follow the two-commit rule).
Methods of Bytes named like deflated/gzipped: snappyCompressed/snappyDecompressed, lz4BlockCompressed/lz4BlockDecompressed(maximumSize), zstdCompressed/zstdDecompressed; failures are the existing CompressionError.
Output of the compressors is a function of the input alone (greedy matching, no timing or machine dependence), like deflated.
Decoders never trust a declared size for allocation and never panic on malformed input.
No new external dependency: implemented from the public format specifications (Snappy's format description, LZ4's block and frame docs, RFC 8878); the BSD reference implementations are only used to produce test vectors.

Acceptance criteria (epic)

Every step above is closed.
Each of the three formats has methods on Bytes, round-trip tests, vectors from the specs and from a reference implementation, and malformed-input tests.
FIRST-PARTY.md gap 26 is marked done and skills/ is regenerated.

Dependencies

None outside the repository. Step 2 depends on step 1 only for the shared files.

## Context Epic for [FIRST-PARTY.md gap 26](https://git.torb.dev/torbscript/language/src/commit/393ef2608e4a60b8ff76d8b6b408951b822d887d/docs/design/FIRST-PARTY.md#L286): zstd, snappy and lz4 in `std/compression`, which MongoDB's wire compression (snappy, zstd) and Kafka (snappy, lz4, zstd) use; optional for both packages. The design record stays the specification; this epic is the slicing. ## Goal `std/compression` reads and writes Snappy, LZ4 and Zstandard next to DEFLATE and gzip, as methods of `Bytes`, in ordinary TorbScript with no native (the precedent of `std/compression`'s DEFLATE and gzip, RELEASE.md section 7.2). Natives for speed are a later, measured decision (benchmark first, then a native under the two-commit rule), not part of the first pass. ## Why it is an epic Three formats with very different weight: the Snappy and LZ4 block formats are a few hundred lines each, the frame formats need new checksums (xxHash32, xxHash64, CRC-32C), and a Zstandard decoder (RFC 8878: frame, literals with Huffman and FSE, sequences, repeat offsets) is far beyond one PR of one or two commits. ## Steps (one issue each, filed just in time when it starts next) - [ ] 1. Snappy and LZ4 block formats: #432 - [ ] 2. xxHash32 and xxHash64 in `std/digest`, then the LZ4 frame format (`Bytes.lz4Framed` / `lz4Unframed`) in `std/compression` - [ ] 3. Zstandard decoder: frames, raw/RLE/compressed blocks, Huffman and FSE, content checksum (`Bytes.zstdDecompressed`) - [ ] 4. Zstandard encoder: deterministic, greedy, predefined FSE tables, raw and RLE blocks as fallback (`Bytes.zstdCompressed`) - [ ] 5. Snappy stream framings (the CRC-32C framing format and the xerial/Kafka block framing); `gap 26` marked done Steps 2 to 5 are listed here so the order is recorded; they become issues only when they start next. Step 3 may be split further (e.g. literals section first) when it is filed. ## Design points (decided by the loop, see the comment of 2026-10-23) - Pure TorbScript in `std/compression`, one file per format; no native, no syntax change, no new driver flag: **touches the seed: no** (no step needs it; a native added later for performance would follow the two-commit rule). - Methods of `Bytes` named like `deflated`/`gzipped`: `snappyCompressed`/`snappyDecompressed`, `lz4BlockCompressed`/`lz4BlockDecompressed(maximumSize)`, `zstdCompressed`/`zstdDecompressed`; failures are the existing `CompressionError`. - Output of the compressors is a function of the input alone (greedy matching, no timing or machine dependence), like `deflated`. - Decoders never trust a declared size for allocation and never panic on malformed input. - No new external dependency: implemented from the public format specifications (Snappy's format description, LZ4's block and frame docs, RFC 8878); the BSD reference implementations are only used to produce test vectors. ## Acceptance criteria (epic) - [ ] Every step above is closed. - [ ] Each of the three formats has methods on `Bytes`, round-trip tests, vectors from the specs and from a reference implementation, and malformed-input tests. - [ ] FIRST-PARTY.md gap 26 is marked done and `skills/` is regenerated. ## Dependencies None outside the repository. Step 2 depends on step 1 only for the shared files.

orion added this to the Milestone 10: breadth of the standard library milestone

2026-10-05 21:07:22 +00:00

orion added the

loop

label

2026-10-05 21:07:22 +00:00

orion commented

2026-10-09 05:40:23 +00:00

Author

Collaborator

[loop:torbscript] intake: in scope (author orion, label loop), source FIRST-PARTY.md gap 26 (optional), not a duplicate. No dependencies. next: waiting (refine queued, ready buffer full) until 2026-10-10T04:00:00Z

orion commented

2026-10-23 14:22:23 +00:00

Author

Collaborator

[loop:torbscript] claim refine by session:loop-torbscript-20261023 until 2026-10-23T16:22Z

orion referenced this issue

2026-10-23 14:23:48 +00:00

compression: Snappy and LZ4 block formats in std/compression (step 1 of #121) #432

orion commented

2026-10-23 14:24:07 +00:00

Author

Collaborator

[loop:torbscript] decided by the loop: #121 is an epic and is not ready; its first step is #432 (Snappy and LZ4 block formats), labelled loop and ready.
Why: three formats of very different weight. Snappy and LZ4 blocks are a few hundred lines each; the frames need xxHash32/64 and CRC-32C, which std/digest does not have; a Zstandard decoder (RFC 8878: Huffman and FSE, sequences) alone is far beyond one PR of one or two commits. Steps 2 to 5 are recorded in the checklist and are filed just in time.
Decisions: (1) pure TorbScript in std/compression like DEFLATE and gzip (RELEASE.md 7.2), no native; speed natives only after a measurement, under the two-commit rule. (2) Names follow deflated/gzipped (participle methods of Bytes, #146): snappyCompressed, lz4BlockCompressed, zstdCompressed and their ...Decompressed partners; LZ4's block decoder takes maximumSize because the block does not store its size. (3) Licences of the reference algorithms (BSD) are a design input only: implemented from the public format descriptions, nothing copied or linked, no new external dependency, so no owner question. (4) Seed: untouched in every step. (5) No duplicates among the open issues (keyword search: compression, zstd, snappy, lz4, xxhash).

[loop:torbscript] decided by the loop: #121 is an epic and is not `ready`; its first step is #432 (Snappy and LZ4 block formats), labelled `loop` and `ready`. Why: three formats of very different weight. Snappy and LZ4 blocks are a few hundred lines each; the frames need xxHash32/64 and CRC-32C, which std/digest does not have; a Zstandard decoder (RFC 8878: Huffman and FSE, sequences) alone is far beyond one PR of one or two commits. Steps 2 to 5 are recorded in the checklist and are filed just in time. Decisions: (1) pure TorbScript in `std/compression` like DEFLATE and gzip (RELEASE.md 7.2), no native; speed natives only after a measurement, under the two-commit rule. (2) Names follow `deflated`/`gzipped` (participle methods of `Bytes`, #146): `snappyCompressed`, `lz4BlockCompressed`, `zstdCompressed` and their `...Decompressed` partners; LZ4's block decoder takes `maximumSize` because the block does not store its size. (3) Licences of the reference algorithms (BSD) are a design input only: implemented from the public format descriptions, nothing copied or linked, no new external dependency, so no owner question. (4) Seed: untouched in every step. (5) No duplicates among the open issues (keyword search: compression, zstd, snappy, lz4, xxhash).

orion commented

2026-10-23 14:24:07 +00:00

Author

Collaborator

[loop:torbscript] refine: #121 stays an epic (body rewritten: goal, five steps, design points, seed statement, acceptance criteria); step 1 is filed as #432 and set ready. Seed untouched. Tokens: about 120k (sonnet, estimate). next: waiting (epic waits on its steps)

[loop:torbscript] refine: #121 stays an epic (body rewritten: goal, five steps, design points, seed statement, acceptance criteria); step 1 is filed as #432 and set `ready`. Seed untouched. Tokens: about 120k (sonnet, estimate). next: waiting (epic waits on its steps)

orion referenced this issue from a commit

2026-10-23 14:47:26 +00:00

feat(std): snappy and lz4 blocks in std/compression

orion referenced this issue

2026-10-23 14:47:37 +00:00

feat(std): snappy and lz4 blocks in std/compression #433