language server: semantic tokens without the resolver over the whole file per request

orion commented

2026-10-08 01:20:47 +00:00

Collaborator

Context

LANGUAGE-SERVER.md section 15, second bullet: the semantic tokens of a large document are the resolver's over the whole file on every request, about 100 ms in the largest file of the compiler. Split from #10.

Goal

textDocument/semanticTokens/full stops resolving the whole file per request. Two reuses, in this order:

A request for a text the server already answered (same document version) is answered from a kept answer. Editors ask after open, after every change and after diagnostics; the repeats cost nothing.
After a body-only change (the view has a mapping, section 7), the highlight resolver (compiler/src/highlight/resolve.trb, resolveFile) runs only over the top-level statements whose text changed. The tokens of every other statement are the earlier text's, moved through the mapping, as the base's resolutions already are. A statement's tokens depend on the file's global names (fileScopeOf, the outline) and on nothing else of the file, and a body-only edit leaves the outline alone; where the outline changed, there is no mapping and the whole file is resolved.

Design record points (pinned)

Section 12: the tokens are torb highlight's, sharpened by the checker, in the legend the extension always declared; the legend and the encoding do not change.
Section 7: a view answers in the coordinates of the current text; the base's data outside the changed bodies is moved through the mapping, the overlay's inside.
compiler/src/highlight/resolve.trb header: the resolver reads the FileScope and a stack of local scopes and nothing else, which is what makes a per-statement walk the same as the whole-file walk (locals are empty at the file level).

Requirements

The answer for any text is byte-identical to today's. This is a pure cost change: no capability is added (no range, no delta; the server announces full only, as now). Adding either is a protocol change for a later issue.
If time remains after the resolver's share is removed in the 100 ms (the checker-resolution sharpening and the encoding in tokens.trb are also per request and per file), the PR says where the rest goes and leaves it in section 15 as what is left rather than inventing more caches.

Acceptance criteria

Two semanticTokens/full requests for one version run the resolver once; a test counts resolver walks (a counter the test reads) and compares both answers.
After a change inside one function body of a file with many declarations, the resolver walks only the statement(s) that changed; a test counts walked statements, and the answer equals the answer of a fresh whole-file resolution of the same text (differential test over at least three edits: a local renamed, a call added, a string with a multi-byte character before the edit).
A change that alters the outline (a declaration added, a use changed) resolves the whole file, with a test; so does the first request of a document.
The deprecated modifiers (deprecatedSpansOf), the sharpened members and the existing token tests (compiler/tests/language-server.test.trb "five integers per token", "a deprecated declaration ...", "hover and tokens inside and outside of a changed body", tests/lsp/features.expected) pass unchanged.
The request is timed on compiler/src/semantics/checker/expression.trb (largest file of the compiler) for a repeated request and for a request after a body-only keystroke, measured on a binary built by the forge CI (not a gate). The numbers are written into section 12, and the second bullet of section 15 is removed or reduced to what the measurement shows is left.
If the docs change reaches skills/, it is regenerated.

Seed

No. Existing syntax, no native, no flag.

Size and commits

One PR, one or two Conventional Commits: perf(lsp): answer semantic tokens from what the last request resolved, and optionally docs(lsp): measure the semantic tokens. Tier A in the forge CI.

Implementer

opus / high (language server core and the highlight resolver; a wrong cache is a wrong color nobody can reproduce, so the differential test is part of the work).

Collisions

Touches compiler/src/language-server/tokens.trb, view.trb or server.trb (the request), and maybe compiler/src/highlight/resolve.trb. #10 touches view.trb and analysis.trb: second to land rebases. No overlap with the running #6, #7, #9.

## Context [LANGUAGE-SERVER.md section 15, second bullet](https://git.torb.dev/torbscript/language/src/commit/068e52da9182672d56447e775c6705661eca5af2/docs/design/LANGUAGE-SERVER.md#L401): the semantic tokens of a large document are the resolver's over the whole file on every request, about 100 ms in the largest file of the compiler. Split from #10. ## Goal `textDocument/semanticTokens/full` stops resolving the whole file per request. Two reuses, in this order: 1. A request for a text the server already answered (same document version) is answered from a kept answer. Editors ask after open, after every change and after diagnostics; the repeats cost nothing. 2. After a body-only change (the view has a mapping, section 7), the highlight resolver (`compiler/src/highlight/resolve.trb`, `resolveFile`) runs only over the top-level statements whose text changed. The tokens of every other statement are the earlier text's, moved through the mapping, as the base's resolutions already are. A statement's tokens depend on the file's global names (`fileScopeOf`, the outline) and on nothing else of the file, and a body-only edit leaves the outline alone; where the outline changed, there is no mapping and the whole file is resolved. ## Design record points (pinned) - Section 12: the tokens are `torb highlight`'s, sharpened by the checker, in the legend the extension always declared; the legend and the encoding do not change. - Section 7: a view answers in the coordinates of the current text; the base's data outside the changed bodies is moved through the mapping, the overlay's inside. - `compiler/src/highlight/resolve.trb` header: the resolver reads the `FileScope` and a stack of local scopes and nothing else, which is what makes a per-statement walk the same as the whole-file walk (locals are empty at the file level). ## Requirements - The answer for any text is byte-identical to today's. This is a pure cost change: no capability is added (no `range`, no `delta`; the server announces `full` only, as now). Adding either is a protocol change for a later issue. - If time remains after the resolver's share is removed in the 100 ms (the checker-resolution sharpening and the encoding in `tokens.trb` are also per request and per file), the PR says where the rest goes and leaves it in section 15 as what is left rather than inventing more caches. ## Acceptance criteria - [ ] Two `semanticTokens/full` requests for one version run the resolver once; a test counts resolver walks (a counter the test reads) and compares both answers. - [ ] After a change inside one function body of a file with many declarations, the resolver walks only the statement(s) that changed; a test counts walked statements, and the answer equals the answer of a fresh whole-file resolution of the same text (differential test over at least three edits: a local renamed, a call added, a string with a multi-byte character before the edit). - [ ] A change that alters the outline (a declaration added, a `use` changed) resolves the whole file, with a test; so does the first request of a document. - [ ] The deprecated modifiers (`deprecatedSpansOf`), the sharpened members and the existing token tests (`compiler/tests/language-server.test.trb` "five integers per token", "a deprecated declaration ...", "hover and tokens inside and outside of a changed body", `tests/lsp/features.expected`) pass unchanged. - [ ] The request is timed on `compiler/src/semantics/checker/expression.trb` (largest file of the compiler) for a repeated request and for a request after a body-only keystroke, measured on a binary built by the forge CI (not a gate). The numbers are written into section 12, and the second bullet of section 15 is removed or reduced to what the measurement shows is left. - [ ] If the docs change reaches `skills/`, it is regenerated. ## Seed No. Existing syntax, no native, no flag. ## Size and commits One PR, one or two Conventional Commits: `perf(lsp): answer semantic tokens from what the last request resolved`, and optionally `docs(lsp): measure the semantic tokens`. Tier A in the forge CI. ## Implementer opus / high (language server core and the highlight resolver; a wrong cache is a wrong color nobody can reproduce, so the differential test is part of the work). ## Collisions Touches `compiler/src/language-server/tokens.trb`, `view.trb` or `server.trb` (the request), and maybe `compiler/src/highlight/resolve.trb`. #10 touches `view.trb` and `analysis.trb`: second to land rebases. No overlap with the running #6, #7, #9.

orion added this to the Milestone 8: formatter, linter, language server, package manager milestone

2026-10-08 01:20:47 +00:00

orion added the

loop

label

2026-10-08 01:20:47 +00:00

orion referenced this issue

2026-10-08 01:22:31 +00:00

language server: reuse the declaration sites and the implementation index across keystrokes #10

orion added the

ready

label

2026-10-08 01:22:32 +00:00

orion commented

2026-10-08 01:22:40 +00:00

Author

Collaborator

[loop:torbscript] refine: split from #10 and made ready (acceptance criteria, pinned records, seed: no). Implementer: opus/high. Collides only with #10 on view.trb (second to land rebases), next: implement