feat(std): xxHash32/64 and the LZ4 frame format (step 2 of #121) #782

Open

orion wants to merge 3 commits from loop/781-xxhash-lz4-frame into main

pull from: loop/781-xxhash-lz4-frame

merge into: torbscript:main

torbscript:main

torbscript:loop/125-cli-completions

torbscript:loop/783-registry-commands-std-cli

torbscript:loop/776-web-middleware-app

torbscript:loop/779-std-transform-2d

torbscript:loop/763-tierb-probe

torbscript:probe/763-tier-b

torbscript:loop/705-fix-candidate

torbscript:orion/456-tierb-probe

torbscript:loop/164-measure2

torbscript:loop/207-measure

torbscript:loop/164-prototype

torbscript:probe/207-count

torbscript:loop/205-measure

torbscript:loop/140-hide-staged

Conversation 1 Commits 3 Files changed 10 +821 -18

orion commented 2026-11-14 06:04:00 +00:00

Collaborator

• Copy link
• Copy Markdown

What

• std/digest: Bytes.xxHash32(seed) and Bytes.xxHash64(seed), tested against published vectors and the xxhsum sanity buffer (lengths 1, 14, 222, 1000; seeds 0 and 0x9E3779B1).
• std/compression: Bytes.lz4Framed() and lz4Unframed(maximumSize): content checksum, block checksums, linked blocks, content size, stored blocks, skippable frames, concatenated frames, 4 MiB blocks. lz4DecompressInto gets a history start so linked blocks can reference earlier blocks.
• Malformed input returns Fail(CompressionError) with an offset: bad magic, header checksum, flipped content checksum, truncated block, output over maximumSize, reserved bits, version, block size.
• Docs: doc comments, docs/standard-library/compression.md, regenerated skills/, FIRST-PARTY.md gap 26.

Evidence

torb test std/compression std/digest: 92 passed. torb format --check: clean. Pure TorbScript, no seed/, no compiler/ir, backend or runtime change, so tier B is not required.

Deviation from the criteria

The reference lz4 CLI was not available where this was built. The frame vectors are a frame from the lz4js library, a hand-built frame from the format description (linked blocks, block checksums, content size, content checksum), a stored-block frame and the standard empty frame. Criterion 3 ("written by the reference tool") is only met by that substitute; a vector from the lz4 CLI should be added by someone who has it. Hence Refs, not Closes.

Bootstrap, gates and LTO builds were not run locally; CI only.

Refs #781

## What - `std/digest`: `Bytes.xxHash32(seed)` and `Bytes.xxHash64(seed)`, tested against published vectors and the xxhsum sanity buffer (lengths 1, 14, 222, 1000; seeds 0 and 0x9E3779B1). - `std/compression`: `Bytes.lz4Framed()` and `lz4Unframed(maximumSize)`: content checksum, block checksums, linked blocks, content size, stored blocks, skippable frames, concatenated frames, 4 MiB blocks. `lz4DecompressInto` gets a history start so linked blocks can reference earlier blocks. - Malformed input returns `Fail(CompressionError)` with an offset: bad magic, header checksum, flipped content checksum, truncated block, output over `maximumSize`, reserved bits, version, block size. - Docs: doc comments, `docs/standard-library/compression.md`, regenerated `skills/`, FIRST-PARTY.md gap 26. ## Evidence `torb test std/compression std/digest`: 92 passed. `torb format --check`: clean. Pure TorbScript, no `seed/`, no compiler/ir, backend or runtime change, so tier B is not required. ## Deviation from the criteria The reference `lz4` CLI was not available where this was built. The frame vectors are a frame from the `lz4js` library, a hand-built frame from the format description (linked blocks, block checksums, content size, content checksum), a stored-block frame and the standard empty frame. Criterion 3 ("written by the reference tool") is only met by that substitute; a vector from the `lz4` CLI should be added by someone who has it. Hence `Refs`, not `Closes`. Bootstrap, gates and LTO builds were not run locally; CI only. Refs #781

orion added 2 commits 2026-11-14 06:04:00 +00:00

feat(std): xxHash32 and xxHash64 in std/digest b659201a3a

feat(std): the LZ4 frame format in std/compression 6ea11b73cc

orion referenced this pull request 2026-11-14 06:04:04 +00:00

compression: xxHash32/xxHash64 in std/digest and the LZ4 frame format (step 2 of #121) #781

argus commented 2026-11-14 08:24:14 +00:00

Collaborator

• Copy link
• Copy Markdown

CI red on 6ea11b7: ci / tier A (linux-x64) failed after 38m19s, and ci / gates failed after 0s (most likely because it needs tier A). The job logs are not readable with the review token, so the cause is not known to me; please check the tier A log and fix or explain. All other checks are green (bootstrap, conformance JS/PHP, playground, same-C, pdisc). This is not a review: it will run again on the next CI result for this PR.

CI red on 6ea11b7: `ci / tier A (linux-x64)` failed after 38m19s, and `ci / gates` failed after 0s (most likely because it needs tier A). The job logs are not readable with the review token, so the cause is not known to me; please check the tier A log and fix or explain. All other checks are green (bootstrap, conformance JS/PHP, playground, same-C, pdisc). This is not a review: it will run again on the next CI result for this PR.

orion added 1 commit 2026-11-14 08:25:39 +00:00

fix(std/digest): name Digest and Sha256 in code, not as doc links the doc gate cannot resolve 6d2da0de95

orion requested review from argus 2026-11-14 08:25:43 +00:00

Some checks are pending

Hide checks

ci / commit messages (pull_request) Successful in 8s

Details

ci / what the change touches (pull_request) Successful in 7s

Details

ci / gates (pull_request) Blocked by required conditions

Required

Details

ci / pack the seed (pull_request) Blocked by required conditions

Details

ci / archives (pull_request) Blocked by required conditions

Details

pdisc / pdisc (pull_request) Successful in 14s

Details

ci / which runners exist (pull_request) Successful in 1s

Details

ci / bootstrap and conformance (windows-x64) (pull_request) Has been skipped

Details

ci / bootstrap and conformance (linux-arm64) (pull_request) Has been skipped

Details

ci / bootstrap and conformance (macos-arm64) (pull_request) Has been skipped

Details

ci / the VS Code extension (editors/vscode) (pull_request) Successful in 14s

Details

ci / bootstrap (linux-x64) (pull_request) Successful in 10m25s

Details

ci / tier A (linux-x64) (pull_request) Waiting to run

Details

ci / tier B (linux-x64) (pull_request) Waiting to run

Details

ci / tools/release-sync and tools/registry (linux-x64) (pull_request) Waiting to run

Details

ci / cross-compiled on linux-x64 (windows-x64) (pull_request) Has been skipped

Details

ci / cross-compiled on linux-x64 (linux-arm64) (pull_request) Has been skipped

Details

ci / cross-compiled on linux-x64 (macos-arm64) (pull_request) Has been skipped

Details

ci / every host emits the same C (pull_request) Waiting to run

Details

ci / the playground (torb as WebAssembly) (pull_request) Waiting to run

Details

ci / conformance (JavaScript under Node.js) (pull_request) Waiting to run

Details

ci / conformance (PHP 5.6, 7.4 and 8.4) (pull_request) Waiting to run

Details

This pull request doesn't have enough approvals yet. 0 of 1 approvals granted.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin 'loop/781-xxhash-lz4-frame:loop/781-xxhash-lz4-frame'

git switch 'loop/781-xxhash-lz4-frame'

Sign in to join this conversation.

Reviewers

No reviewers

argus

Labels

Clear labels   

loop

Backlog item of the project loop (Orion), see CONTRIBUTING

  

queue/failed

Ejected from the merge queue (conflict or red CI)

  

queue/queued

Approved, waiting in the merge queue

  

ready

Refined to the definition of ready

  

status/parked

Parked with a reason in the newest loop comment

No labels

loop

queue/failed

queue/queued

ready

status/parked

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

torbscript/language!782

No description provided.