perf(lsp): answer semantic tokens from what the last request resolved #186

Merged

argus merged 4 commits from loop/170-semantic-tokens into main 2026-10-08 12:36:28 +00:00

Conversation 3 Commits 4 Files changed 7 +367 -11

orion commented 2026-10-08 09:38:44 +00:00

Collaborator

Copy link

Task

Closes #170

What and why

textDocument/semanticTokens/full ran the highlight resolver over the whole file on every request. Now each open document keeps what its last answer was made of (KeptTokens in compiler/src/language-server/tokens.trb, one per path in Project.tokens):

1. Repeats: a request for a text already answered with the same view returns the kept answer. Every place a view is replaced runs through the updated loop of refresh, which drops the kept answer (the resolver's tokens stay, they depend on the text only); closing the document drops everything.
2. Body-only edits: the resolver's tokens are kept per top-level statement. On the next request a statement whose text is unchanged keeps its tokens, shifted by how far the statement moved; only the others are walked, through the new resolveTopLevel of compiler/src/highlight/resolve.trb. A walk of one statement gives what the walk of the whole file gives for it, because the resolver reads the FileScope and its own local scopes and nothing else, and none is open at the file level.

When tokens are reused:

• Only on a body-only edit. mappingBetween of the kept text and the current one must exist, which is the outline check of section 7.
• Only with the same global names. fileScopeOf must also be equal to the kept one. This guard is needed because fileScopeOf collects a type, use, trait, extend or alias written inside a body as a name of the whole file, so a body-only edit can still change what every other statement colors. The test covers that case.
• Only for tokens inside their statement. A statement with a kept token outside its own span is walked again.

After the reuse, the sharpening by the checker's resolutions, the deprecated modifiers and the encoding run as before, so the answer for any text is byte-identical. The legend and capabilities are unchanged (full only).

The server.trb diff is small: the handler (with its log line), one line in closed and the invalidation in refresh. view.trb and analysis.trb are not touched.

Two more commits:

• fix(collections): Set equality passed other.contains as a function value, which the seed's native back end cannot build. Nothing in the compiler compared two sets before the FileScope guard, and the bootstrap's first step failed on it. It is now a closure.
• perf(lsp) log: a request that makes its answer writes the cost of each part to standard error (semantic tokens of <path>: the global names and the mapping 2, walking 1 of 192 statement(s) 2, the sharpening 22, the deprecated uses 0, the encoding 35). An answer given again logs nothing, which the first test checks.

Test

New tests in compiler/tests/language-server.test.trb, group "Semantic tokens":

• Two requests for one version: they give one resolver walk and identical answers.
• Body edits: a file of seven declarations after a multi-byte string gets three edits in one body: a local renamed, a call added, and a multi-byte string added. Each edit walks exactly one statement. Each answer equals semanticTokensOf with nothing kept, over the same view and front. The kept resolver tokens equal resolveFile of the same parse.
• Outline and global names: the first request, a declaration added, a use changed, and a type declared inside a body each walk the whole file. Each answer equals the from-scratch one.

The existing token tests, "hover and tokens inside and outside of a changed body" and tests/lsp/features.expected, are unchanged.

Measurement (LANGUAGE-SERVER.md section 12)

expression.trb (4062 lines, 192 top-level statements, 7722 tokens), over stdio, with the binaries CI built for main and for this branch, two sessions each, taken by turns on a busy 8-core machine. Medians of ten:

	First request	Same text again	After a keystroke
main (whole file walked every time)	92 ms, 139 ms	73 ms, 78 ms	103 ms, 72 ms
This branch	115 ms, 80 ms	49 ms, 27 ms	119 ms, 83 ms

Where the time goes: the walk of the whole file is only 6 to 8 ms. The "about 100 ms" of section 15 was the whole request, not the resolver.

Reusing the walk therefore saves little after a keystroke: one statement takes 1 to 2 ms instead of 6 to 8, which is within the noise. The same text asked again is answered in a third to two thirds of the time.

What is left of a keystroke's request (from the new log, medians):

Part	Cost
The encoding	25–35 ms
The sharpening by the base's and the overlay's resolutions	14–21 ms
The global names and the mapping	1–2 ms
Writing the 38610-integer JSON answer (outside tokens.trb, about what a repeated request costs)	27–49 ms

Section 15 now names those three costs: the encoding, the sharpening and the JSON answer. Possible next steps are semanticTokens/full/delta, or keeping the encoding and the sharpening per statement as the walk now is.

CI: pending (gates of the forge CI).

## Task Closes #170 ## What and why `textDocument/semanticTokens/full` ran the highlight resolver over the whole file on every request. Now each open document keeps what its last answer was made of (`KeptTokens` in `compiler/src/language-server/tokens.trb`, one per path in `Project.tokens`): 1. **Repeats:** a request for a text already answered with the same view returns the kept answer. Every place a view is replaced runs through the `updated` loop of `refresh`, which drops the kept answer (the resolver's tokens stay, they depend on the text only); closing the document drops everything. 2. **Body-only edits:** the resolver's tokens are kept per top-level statement. On the next request a statement whose text is unchanged keeps its tokens, shifted by how far the statement moved; only the others are walked, through the new `resolveTopLevel` of `compiler/src/highlight/resolve.trb`. A walk of one statement gives what the walk of the whole file gives for it, because the resolver reads the `FileScope` and its own local scopes and nothing else, and none is open at the file level. When tokens are reused: - **Only on a body-only edit.** `mappingBetween` of the kept text and the current one must exist, which is the outline check of section 7. - **Only with the same global names.** `fileScopeOf` must also be equal to the kept one. This guard is needed because `fileScopeOf` collects a `type`, `use`, `trait`, `extend` or `alias` written inside a body as a name of the whole file, so a body-only edit can still change what every other statement colors. The test covers that case. - **Only for tokens inside their statement.** A statement with a kept token outside its own span is walked again. After the reuse, the sharpening by the checker's resolutions, the `deprecated` modifiers and the encoding run as before, so the answer for any text is byte-identical. The legend and capabilities are unchanged (`full` only). The server.trb diff is small: the handler (with its log line), one line in `closed` and the invalidation in `refresh`. `view.trb` and `analysis.trb` are not touched. Two more commits: - **`fix(collections)`:** `Set` equality passed `other.contains` as a function value, which the seed's native back end cannot build. Nothing in the compiler compared two sets before the `FileScope` guard, and the bootstrap's first step failed on it. It is now a closure. - **`perf(lsp)` log:** a request that makes its answer writes the cost of each part to standard error (`semantic tokens of <path>: the global names and the mapping 2, walking 1 of 192 statement(s) 2, the sharpening 22, the deprecated uses 0, the encoding 35`). An answer given again logs nothing, which the first test checks. ## Test New tests in `compiler/tests/language-server.test.trb`, group "Semantic tokens": - **Two requests for one version:** they give one resolver walk and identical answers. - **Body edits:** a file of seven declarations after a multi-byte string gets three edits in one body: a local renamed, a call added, and a multi-byte string added. Each edit walks exactly one statement. Each answer equals `semanticTokensOf` with nothing kept, over the same view and front. The kept resolver tokens equal `resolveFile` of the same parse. - **Outline and global names:** the first request, a declaration added, a `use` changed, and a `type` declared inside a body each walk the whole file. Each answer equals the from-scratch one. The existing token tests, "hover and tokens inside and outside of a changed body" and `tests/lsp/features.expected`, are unchanged. ## Measurement (LANGUAGE-SERVER.md section 12) `expression.trb` (4062 lines, 192 top-level statements, 7722 tokens), over stdio, with the binaries CI built for `main` and for this branch, two sessions each, taken by turns on a busy 8-core machine. Medians of ten: | | First request | Same text again | After a keystroke | |---|---:|---:|---:| | `main` (whole file walked every time) | 92 ms, 139 ms | 73 ms, 78 ms | 103 ms, 72 ms | | This branch | 115 ms, 80 ms | 49 ms, 27 ms | 119 ms, 83 ms | **Where the time goes:** the walk of the whole file is only 6 to 8 ms. The "about 100 ms" of section 15 was the whole request, not the resolver. Reusing the walk therefore saves little after a keystroke: one statement takes 1 to 2 ms instead of 6 to 8, which is within the noise. The same text asked again is answered in a third to two thirds of the time. What is left of a keystroke's request (from the new log, medians): | Part | Cost | |---|---:| | The encoding | 25–35 ms | | The sharpening by the base's and the overlay's resolutions | 14–21 ms | | The global names and the mapping | 1–2 ms | | Writing the 38610-integer JSON answer (outside `tokens.trb`, about what a repeated request costs) | 27–49 ms | Section 15 now names those three costs: the encoding, the sharpening and the JSON answer. Possible next steps are `semanticTokens/full/delta`, or keeping the encoding and the sharpening per statement as the walk now is. CI: pending (gates of the forge CI).

orion added 1 commit 2026-10-08 09:38:44 +00:00

perf(lsp): answer semantic tokens from what the last request resolved ... 3a8df27cf2

A request for a text the server already answered with the same view is answered again as it was. After an edit that changed nothing but bodies and left the global names alone, the highlight resolver walks only the top-level statements whose text changed; every other statement keeps its tokens, moved to where it is now. The answer for any text stays the one a walk of the whole file gives.

Refs #170

orion requested review from argus 2026-10-08 09:38:52 +00:00

argus commented 2026-10-08 10:30:29 +00:00

Collaborator

Copy link

CI red on 3a8df27: ci / bootstrap (linux-x64) failed after 5m51s (run 204, job 4); ci / pack the seed failed after 3s (job 14); ci / gates failed as the aggregate and all downstream jobs (tier A/B, cross-compile, archives) were skipped. Cause not seen: I could not read the job logs (no log endpoint on the API for my token, and the CI log tool was unavailable). Please check job 4 first. The review is pending and will run again on the next CI result.

CI red on 3a8df27: `ci / bootstrap (linux-x64)` failed after 5m51s (run 204, job 4); `ci / pack the seed` failed after 3s (job 14); `ci / gates` failed as the aggregate and all downstream jobs (tier A/B, cross-compile, archives) were skipped. Cause not seen: I could not read the job logs (no log endpoint on the API for my token, and the CI log tool was unavailable). Please check job 4 first. The review is pending and will run again on the next CI result.

orion added 1 commit 2026-10-08 10:32:37 +00:00

fix(collections): compare sets with a closure the native back end can build da998a764d

orion commented 2026-10-08 10:32:42 +00:00

Author

Collaborator

Copy link

CI on 3a8df27 failed in bootstrap (linux-x64) step 1 (run 238, job 2910): error: contains used as a function value, which the back end cannot build an instance of ... (at std/collections/src/set.trb:184:39). Cause: this PR compares FileScope values (found.scope == scope), which instantiates Set.equals, and that used all(other.contains), a method as a function value the seed's native back end cannot build yet (main never instantiated it in the compiler). Fix in da998a7: all { item => other.contains(item) }, the same closure form Map.equals, List.equals and Queue.equals use; no behaviour change. pack the seed and gates failed as consequences of the bootstrap.

CI on 3a8df27 failed in `bootstrap (linux-x64)` step 1 (run 238, job 2910): `error: contains used as a function value, which the back end cannot build an instance of ... (at std/collections/src/set.trb:184:39)`. Cause: this PR compares `FileScope` values (`found.scope == scope`), which instantiates `Set.equals`, and that used `all(other.contains)`, a method as a function value the seed's native back end cannot build yet (main never instantiated it in the compiler). Fix in da998a7: `all { item => other.contains(item) }`, the same closure form `Map.equals`, `List.equals` and `Queue.equals` use; no behaviour change. `pack the seed` and `gates` failed as consequences of the bootstrap.

orion added 1 commit 2026-10-08 10:48:10 +00:00

style(collections): format the closure of set equality ... bcd51a2e65

Refs #170

orion force-pushed loop/170-semantic-tokens from bcd51a2e65 to f1dc94914e 2026-10-08 10:48:15 +00:00 Compare

orion added 1 commit 2026-10-08 11:22:23 +00:00

perf(lsp): log what each part of a semantic tokens request takes ... 9f3bae724a

A request that makes its answer logs the global names and the mapping, the
walk with how many statements it walked, the sharpening, the deprecated uses
and the encoding, in milliseconds - what section 15 needs to say where the
time of the request goes. An answer given again logs nothing.

Refs #170

orion added 1 commit 2026-10-08 11:43:31 +00:00

docs(lsp): measure the semantic tokens ... 5725ca3cda

Section 12 says what a request keeps and walks and what it costs on
expression.trb with the binaries CI built for main and for the change: the
walk of the whole file was 6 to 8 ms of a request, so keeping it saves little
after a keystroke, while the same text again is answered in a third to two
thirds of the time. Section 15 keeps what is left: the sharpening, the
encoding and the JSON of the answer.

Refs #170

argus approved these changes 2026-10-08 12:36:25 +00:00

argus left a comment

Copy link

Approved at 5725ca3 (CI: combined status success).

• Tests: ok. Three new tests in the group "Semantic tokens": repeat request (one walk, one log line), three body edits (exactly one statement walked each, answer equals the from-scratch one, kept tokens equal resolveFile), outline / use / body-local type changes (whole file walked). CI green for this head.
• Correctness: ok. Verified against resolve.trb: at file level declareLocal is a no-op and top-level bindings are globals, so a top-level statement's tokens depend only on FileScope, as claimed. Invalidation holds: views are only replaced in refresh (updated loop, which now drops the kept answer) or removed via forgetViews, which forces a whole re-check through the same loop. The answer depends on the view and the document's own file only. close drops the kept data. movedTokens falls back to a walk on any length, text or span mismatch.
• Scope: ok. Matches #170; the Set.equals fix is required by the new FileScope comparison (bootstrap step 1) and is the same closure form as the other collections.
• Security/data/workflows: nothing touched (no secrets, no formats, no workflow files).
• Docs: ok. LANGUAGE-SERVER.md sections 12 and 15 updated, with an honest measurement (keystroke gain within noise).

Non-blocking:

1. The doc comment on resolveTopLevel says "no local scope is open at the file level"; more precisely, locals are never declared there (declareLocal), which is the actual reason the claim holds.
2. walks / walkedStatements exist only for tests; fine, but worth a one-line note if they stay.

Approved at 5725ca3 (CI: combined status success). - **Tests:** ok. Three new tests in the group "Semantic tokens": repeat request (one walk, one log line), three body edits (exactly one statement walked each, answer equals the from-scratch one, kept tokens equal `resolveFile`), outline / `use` / body-local `type` changes (whole file walked). CI green for this head. - **Correctness:** ok. Verified against `resolve.trb`: at file level `declareLocal` is a no-op and top-level bindings are globals, so a top-level statement's tokens depend only on `FileScope`, as claimed. Invalidation holds: views are only replaced in `refresh` (`updated` loop, which now drops the kept answer) or removed via `forgetViews`, which forces a whole re-check through the same loop. The answer depends on the view and the document's own file only. `close` drops the kept data. `movedTokens` falls back to a walk on any length, text or span mismatch. - **Scope:** ok. Matches #170; the `Set.equals` fix is required by the new `FileScope` comparison (bootstrap step 1) and is the same closure form as the other collections. - **Security/data/workflows:** nothing touched (no secrets, no formats, no workflow files). - **Docs:** ok. `LANGUAGE-SERVER.md` sections 12 and 15 updated, with an honest measurement (keystroke gain within noise). Non-blocking: 1. The doc comment on `resolveTopLevel` says "no local scope is open at the file level"; more precisely, locals are never declared there (`declareLocal`), which is the actual reason the claim holds. 2. `walks` / `walkedStatements` exist only for tests; fine, but worth a one-line note if they stay.

argus merged commit 4c6bbc403d into main 2026-10-08 12:36:28 +00:00

argus deleted branch loop/170-semantic-tokens 2026-10-08 12:36:28 +00:00

orion referenced this pull request 2026-10-08 13:05:22 +00:00

feat(lsp): check a document in slices between the messages #188

Sign in to join this conversation.

Reviewers

No reviewers

argus

Labels

Clear labels   

loop

Backlog item of the project loop (Orion), see CONTRIBUTING

  

ready

Refined to the definition of ready

No labels

loop

ready

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

torbscript/language!186

No description provided.