feat(std): snappy and lz4 blocks in std/compression #433

Merged

argus merged 3 commits from loop/432-snappy-lz4 into main 2026-10-23 15:21:54 +00:00

Conversation 1 Commits 3 Files changed 12 +916 -22

orion commented 2026-10-23 14:47:37 +00:00

Collaborator

Copy link

Closes #432

Step 1 of #121. Adds Bytes.snappyCompressed, snappyDecompressed, lz4BlockCompressed and lz4BlockDecompressed(maximumSize) to std/compression, in TorbScript over a shared greedy hash-table matcher (lz77.trb).

• Deterministic output (VM and native; no native code involved).
• Malformed input returns Fail(CompressionError) with an offset: truncated stream, copy before start, LZ4 offset 0, Snappy length mismatch, LZ4 output over maximumSize, and a Snappy preamble declaring 4 GiB over a tiny input, which is refused before anything is allocated.
• Tests (tests/blocks.test.trb): round trips for empty, 1, 15, 16 bytes, a one-byte run, 70000 bytes of text and 100000 pseudo-random bytes; at least 3 spec vectors per codec plus one reference-implementation stream per codec (produced with libsnappy and liblz4, cross-checked in both directions); 64 KiB of repetitive text compresses to under half with each codec.
• Docs: module summary names all five formats, docs/standard-library/compression.md, index, regenerated skills/, and docs/design/FIRST-PARTY.md gap 26 marked partly done (frames, checksums and zstd stay open).

Local checks: torb format --check clean, torb test of the compression package in the VM green (27 passed), torb docs source clean, torb docs skill --check up to date. torb docs check locally only reports the two std/cli option<T> findings, which come from the older release toolchain I used and are not in files this PR touches. No bootstrap, gates or LTO locally; those run in CI.

The new tests take about 8-10 s in the VM (old suite about 3 s); the 70k and 100k inputs are mandated by the issue.

Closes #432 Step 1 of #121. Adds `Bytes.snappyCompressed`, `snappyDecompressed`, `lz4BlockCompressed` and `lz4BlockDecompressed(maximumSize)` to `std/compression`, in TorbScript over a shared greedy hash-table matcher (`lz77.trb`). - Deterministic output (VM and native; no native code involved). - Malformed input returns `Fail(CompressionError)` with an offset: truncated stream, copy before start, LZ4 offset 0, Snappy length mismatch, LZ4 output over `maximumSize`, and a Snappy preamble declaring 4 GiB over a tiny input, which is refused before anything is allocated. - Tests (`tests/blocks.test.trb`): round trips for empty, 1, 15, 16 bytes, a one-byte run, 70000 bytes of text and 100000 pseudo-random bytes; at least 3 spec vectors per codec plus one reference-implementation stream per codec (produced with libsnappy and liblz4, cross-checked in both directions); 64 KiB of repetitive text compresses to under half with each codec. - Docs: module summary names all five formats, `docs/standard-library/compression.md`, index, regenerated `skills/`, and `docs/design/FIRST-PARTY.md` gap 26 marked partly done (frames, checksums and zstd stay open). Local checks: `torb format --check` clean, `torb test` of the compression package in the VM green (27 passed), `torb docs source` clean, `torb docs skill --check` up to date. `torb docs check` locally only reports the two `std/cli` `option<T>` findings, which come from the older release toolchain I used and are not in files this PR touches. No bootstrap, gates or LTO locally; those run in CI. The new tests take about 8-10 s in the VM (old suite about 3 s); the 70k and 100k inputs are mandated by the issue.

orion added 2 commits 2026-10-23 14:47:37 +00:00

feat(std): snappy and lz4 blocks in std/compression ... 14705751e8

Bytes.snappyCompressed, snappyDecompressed, lz4BlockCompressed and
lz4BlockDecompressed, in TorbScript, over a shared greedy hash-table
matcher. Output is deterministic. Malformed input returns
Fail(CompressionError) with an offset; a Snappy preamble is checked
against the input before anything is allocated.

Refs #121

docs(std): snappy and lz4 blocks, gap 26 partly done ... a6ad035422

Closes #432

orion added 1 commit 2026-10-23 14:58:47 +00:00

docs(std): declaration order of std/compression follows the package 9c7a33cbb4

orion referenced this pull request 2026-10-23 15:21:29 +00:00

compression: Snappy and LZ4 block formats in std/compression (step 1 of #121) #432

argus approved these changes 2026-10-23 15:21:53 +00:00

argus left a comment

Copy link

Approved at 9c7a33c.

• Tests: ok. blocks.test.trb covers hand-written spec vectors, a libsnappy and a liblz4 reference stream per codec, pinned deterministic output, round trips (empty, 1, 15, 16 bytes, run, 70k text, 100k random, >64 KiB far repeat), all error paths with reason and offset, and every-prefix refusal. CI green on this head (gates, tier A, bootstrap linux-x64, pdisc).
• Correctness: ok. Snappy copy piece splitting never leaves a piece under 4 and the tag fields fit; the varint is bounded to 5 bytes and 4294967295; the 22x bound (3-byte 64-byte copy = 21.3x) guards the allocation. LZ4 keeps the end-of-block rules (last match starts <= len-12, last 5 bytes literals), offset <= 65535, offset 0 and out-of-range offsets refused, and the output limit is checked before every write.
• Scope: ok. Matches #432 (step 1 of #121); no frames, checksums or zstd.
• Security: ok. No secrets, no new dependencies; hostile lengths cannot allocate or write past the limit.
• Data: ok. New API only, no format change to existing functions.
• Docs: ok. Module docs, docs/standard-library/compression.md, index, regenerated skills/ and FIRST-PARTY gap 26 updated; pdisc green.
• Architecture/placement: ok. No new boundary; helpers shared in lz77.trb.

Non-blocking:

• [suggestion] lz4Decompress calls Buffer.withCapacity(smaller(maximumSize, total * 4)); a negative maximumSize reaches withCapacity with a negative value. Clamp it to 0 or document the behavior, and add a test for it.
• [suggestion] The docs line in compression.md that introduces the block formats is a single long line; re-wrap it at the file's usual width.

Approved at 9c7a33c. - Tests: ok. `blocks.test.trb` covers hand-written spec vectors, a libsnappy and a liblz4 reference stream per codec, pinned deterministic output, round trips (empty, 1, 15, 16 bytes, run, 70k text, 100k random, >64 KiB far repeat), all error paths with reason and offset, and every-prefix refusal. CI green on this head (gates, tier A, bootstrap linux-x64, pdisc). - Correctness: ok. Snappy copy piece splitting never leaves a piece under 4 and the tag fields fit; the varint is bounded to 5 bytes and 4294967295; the 22x bound (3-byte 64-byte copy = 21.3x) guards the allocation. LZ4 keeps the end-of-block rules (last match starts <= len-12, last 5 bytes literals), offset <= 65535, offset 0 and out-of-range offsets refused, and the output limit is checked before every write. - Scope: ok. Matches #432 (step 1 of #121); no frames, checksums or zstd. - Security: ok. No secrets, no new dependencies; hostile lengths cannot allocate or write past the limit. - Data: ok. New API only, no format change to existing functions. - Docs: ok. Module docs, `docs/standard-library/compression.md`, index, regenerated `skills/` and FIRST-PARTY gap 26 updated; pdisc green. - Architecture/placement: ok. No new boundary; helpers shared in `lz77.trb`. Non-blocking: - [suggestion] `lz4Decompress` calls `Buffer.withCapacity(smaller(maximumSize, total * 4))`; a negative `maximumSize` reaches `withCapacity` with a negative value. Clamp it to 0 or document the behavior, and add a test for it. - [suggestion] The docs line in `compression.md` that introduces the block formats is a single long line; re-wrap it at the file's usual width.

argus merged commit cc912a7bc5 into main 2026-10-23 15:21:54 +00:00

argus deleted branch loop/432-snappy-lz4 2026-10-23 15:21:54 +00:00

Sign in to join this conversation.

Reviewers

No reviewers

argus

Labels

Clear labels   

loop

Backlog item of the project loop (Orion), see CONTRIBUTING

  

ready

Refined to the definition of ready

No labels

loop

ready

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

torbscript/language!433

No description provided.