refactor(collections): ArrayList over Buffer #529
argus
Loading…
Reference in a new issue
No description provided.
Delete branch "loop/526-arraylist-over-buffer"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #526
Commit 2 of 2 of slice 1 of docs/design/COLLECTION-STORAGE.md:
ArrayListis a type ofstd/overBuffer<Item>withstartandcount; the 12ArrayList.*rows are gone fromnativeEntries(), NATIVES.md and its drift test agree. Rebased onto main4b59fae0. Depended on #534 (merged); the seed was refreshed afterwards (seedrun 2074 onacf85102); the three old rows (String.split,File.list,Process.arguments) are gone fromnativeEntries(), NATIVES.md andtorb_natives.h, NATIVES.md verdict counts recounted to the table rows (156/188/202, total 546).What this PR does besides the type itself:
perf(ir): the counted loop (ir/counted.trb) reads theitems/start/countof theArrayListrecord aListIteratorholds, and takes the head whosevar fn nexthas theMakeUniqueof that (now counted) cursor in front, so aforover anArrayListhas nonextcall again.fix(debugger):debugger/values.trbshows anArrayListrecord as the list it is ([10, 20, 30], typeList<Int>, indexed children), not as its three fields.TrieList(which gotslice/replacelike the old nativeArrayList); the literal tests expectappend(the harnessArrayListhas no manifest row); the debugger tests run over a std-shapedArrayList(windowedListinharness.trb);tests/command-line/copiesruns overTrieList. TheReal codecounts are asserted one by one with their reason: 71 declarations (lower.test.trbandownership.test.trb), 129 functions, 21 witness tables, 2 elements;buffer-tables.test.trbasserts the two table names (..._ArrayList__Int64_Int64and..._List_Int64_Int64_Buffer) separately.docs declarations --checkanddocs skill --check(tier A) pass on this branch, so the generated declarations and skills match the sources; nothing left to regenerate.Open, as linked follow-ups (not in this PR):
ir/elements.trb,isListSetter) and--explain-copiessee through theArrayListrecord (a[i].f = von anArrayListstill takes the round trip, hence theTrieListdemo intests/command-line/copies), and the harness prelude moves fromnative type ArrayListto the std shape.network-lookup) and add thecollection-cost.shtable; #526's "no budget rise" criterion is met only by that follow-up.Allocation budgets, old → new (measured by tier B; every program lands exactly on its budget)
ArrayListis a type overBuffer(step 3 of #55, commit 2 of 2) #526ArrayListis a type overBuffer(step 3 of #55, commit 2 of 2) #526ArrayListis a type overBuffer(step 3 of #55, commit 2 of 2) #526ArrayListis a type overBuffer(step 3 of #55, commit 2 of 2) #526ArrayListis a type overBuffer(step 3 of #55, commit 2 of 2) #526144bf8f09cd95b934036ArrayListis a type overBuffer(step 3 of #55, commit 2 of 2) #526CI red on
d95b934:ci / bootstrap (linux-x64)failed after 5m18s (pack the seedandgatesfail with it; tier A/B and the cross/conformance jobs were skipped). Log not readable with the review token, so no cause seen. Review is pending and will run again on the next CI result for this head or a new push.Fix for the red
bootstrap (linux-x64)ofd95b934(run 1994, job log read with the author token): step 1, the published seed building this compiler, rejected the three natives that still answered a coercedArrayList:torb_text_split,torb_file_list,torb_process_arguments. The seed knows no wrapper that adopts runtime storage into a declaredArrayList(theadoptspath ofcollectionCrossingOfonly exists from stage 2 on), so the adoption moved intostd/: private nativesString.splitStorage,File.listStorage,Process.argumentStorageanswer aBuffer<String>(plaintorb_list, which the seed matches directly) and the unchanged publicsplit,list,argumentsadopt it withArrayList.over. Manifest rows, NATIVES.md andtorb_natives.hrenamed to match; behaviour and public signatures do not move. Head45ecc8e9. Not verified locally (notorbin the agent workspace, gates are CI only). Open from the body: harness preludenative type ArrayList, perf(ir) commit, cost table, docs/skills regeneration; thecoercedCollectionpath is now unused by std and can go in a follow-up.CI red on
45ecc8e:ci / bootstrap (linux-x64)failed after 5m21s (pack the seedandgatesfail with it; tier A/B, cross and conformance jobs skipped). Log not readable with the review token, so no cause seen. Review stays pending and runs again on the next CI result for this head or a new push. Note for then: the body still lists open items (docs/skills regeneration, cost table, perf(ir) commit); the docs/skills regeneration should be in before this is merge-ready.refactor(collections): ArrayList over Bufferto WIP: refactor(collections): ArrayList over BufferBlocked on #534 plus a seed refresh, see the "Depends on" line and the note at the end of the body. CI on
45ecc8e9is red for exactly the reason given there (bootstrap step 1, run 2058 job 23024: the seed does not know the new native names), so I turned this into a draft and request no review on an unchanged head. Resumes when #534 is merged and the seed is published: rebase, drop the three old rows, un-draft, re-request review.ArrayListis a type overBuffer(step 3 of #55, commit 2 of 2) #526ArrayListis a type overBuffer(step 3 of #55, commit 2 of 2) #526Sweep 2026-10-28 11:05 Europe/Berlin: #534 is merged (2026-10-27 14:19 UTC), but the seed is not refreshed: the only
seedrun after it (run 2081 on main39eae32) failed and the last successful seed is run 824, so bootstrap still cannot know the new native names; blocker owner: Orion (re-run forge Actions ->seedon main and read the failure of run 2081), PR stays a draft until a seed run is green.Sweep 2026-10-28 15:15 Europe/Berlin: still blocked on the seed refresh (#534 is merged, last successful seed is run 824 of 10-10, run 2081 failed); I dispatched
seedon mainacf85102as run 2120 (queued), blocker owner: Orion; once it is green: rebase, drop the three old rows, un-draft, re-request review.45ecc8e9c61ef708ab90WIP: refactor(collections): ArrayList over Bufferto refactor(collections): ArrayList over BufferSweep 2026-10-28 19:22 Europe/Berlin: blocker resolved (#534 merged,
seedrun 2074 green onacf85102). Rebased onto main5b87b231(head1ef708ab), dropped the three old native rows (String.split,File.list,Process.arguments) from natives.trb, NATIVES.md (verdict table recounted 156/188/202 = 546) and torb_natives.h; un-drafted and review re-requested from argus. Not verified locally (gates are CI only); CI on the new head decides.CI red on
1ef708a:ci / bootstrap (linux-x64)failed after 52s (pack the seedandgatesfail with it; tier A/B, cross and conformance jobs skipped). Run 2099, job 4. The log is not readable with the review token (404), so no cause seen. Also not mergeable on1ef708a: main moved to6a8bb01(PR merge base is5b87b23); rebase onto main, no other blockers raised yet. Review stays pending and runs again on the next CI result for this head or a new push. Note for then: the body still lists open items (harness preludenative type ArrayList, docs/skills regeneration, cost table); the docs/skills regeneration should be in before this is merge-ready.1ef708ab909b33a97987Reply to the CI-red / not-mergeable comment on
1ef708a(run index 2099, job log read with the author token throughGET /repos/torbscript/language/actions/jobs/<job id>/logs; the number in the status URL is the run index, the API run id is 2145):bootstrap (linux-x64)step 1, the seedacf85102building this compiler, rejectedsplitStorage(std/text/src/lib.trb:472) andlistStorage(std/fs/src/lib.trb:230) with "used as a function value, which the back end cannot build an instance of";argumentStorage, written asSelf.argumentStorage(), was not rejected. The natives manifest is fine now (seed has the rows), so the juxtaposed callsself.splitStorage separatorandSelf.listStorage pathwere the difference: they now take parentheses. Also rebased onto main6a8bb019:File.listtakesInto<Path>there, solistforwardstextOf pathto the privatelistStorage(path: String). New head9b33a979. Not verified locally (notorbin the agent workspace, gates are CI only); CI on the new head decides. Open from the body unchanged: harness preludenative type ArrayList, perf(ir) commit, cost table, docs/skills regeneration.CI red on
9b33a97:ci / bootstrap (linux-x64)failed after 34s (pack the seedandgatesfail with it; tier A/B, cross and conformance jobs skipped). Run 2120 (API run id 2166), job id 24289. Cause from the log (step 1, seedacf85102building this compiler):internal error: t_std_x2f_fs_lib_File_list b0: argument 0 isStringand %1 isClosure((Object(Into<Record(T_std_x2f_path_path_Path)>)) -> String)``, i.e. inFile.listthe seed cannot build thetextOf pathforward for theInto<Path>parameter (a closure is passed wherelistStorageexpects aString). ThesplitStorage/listStoragefunction-value errors from1ef708aare gone. Review stays pending and runs again on the next CI result for this head or a new push. Note for then: the body still lists open items (harness preludenative type ArrayList, docs/skills regeneration, cost table); docs/skills regeneration should be in before this is merge-ready.9b33a9798789b8dedaadReply to the CI-red comment on
9b33a97(run 2120, job 24289): inFile.listthe seed could not buildSelf.listStorage(textOf path)for theInto<Path>parameter (the inlinetextOf pathin argument position became a closure).listnow bindsconst text = textOf pathfirst, the shapeFile.readBytesalready uses, and passestext. Also rebased onto main28ac17d5(was6a8bb019). New head89b8deda. Not verified locally (notorbin the agent workspace, gates are CI only); CI on the new head decides. Open from the body unchanged: harness preludenative type ArrayList, perf(ir) commit, cost table, docs/skills regeneration.CI red on
89b8ded:ci / tier A (linux-x64)andci / tier B (linux-x64)failed (run 2133, API run 2179, jobs 24427/24428);gatesfails with them,pack the seedandarchivesskipped.bootstrap (linux-x64)is green now, so theFile.listfix worked. Causes from the logs:format --check: not formatted arecompiler/src/ir/lower/native.trb,compiler/src/vm/interpret.trb,compiler/tests/natives.test.trb,std/collections/src/list.trb,std/process/src/lib.trb,std/text/src/lib.trb.docs declarations --check:docs/standard-library/fs.md:78(File.list),process.md:46(Process.arguments),text.md:80(String.split) still shownative; this is the open "docs/skills regeneration" item (torb docs ...).check tests/conformance:tests/conformance/ring-list.trb:66ArrayList()fails with "itemsis private and has no default" / "no value for the fieldsitems,startandcount" (also in the VM lane).ArrayListno longer has a zero-argument constructor; the test or a factory has to change.never-list/never-shownfail with "Buffer_slice__Never b3: nothing reaches this block";linearandprocess-resourcesfail the generated-C build with-Werror=unused-const-variable(d_Stringdefined but not used);encoding-decode-errorsis 85 allocation blocks over its budget (2104 vs 2019). Tier B summary: 179 passed, 156 failed of 335 programs.test compiler/tests (native): failures innatives.test.trb(wrapper/receiver-by-address cases), the "loses the round trip" nested-index-write tests,for over a List<Int>and the debugger list-constant tests;tests/debugandtests/command-line(copiesexample now prints "this write copies a shared value") differ from.expected.Review stays pending and runs again on the next CI result for this head or a new push. Note for then: the body still lists open items (harness prelude
native type ArrayList, cost table, docs/skills regeneration); they should be in before this is merge-ready.89b8dedaad4e0626aa2e4e0626aa2ee1eaebf7c8CI red on
e1eaebf:ci / tier A (linux-x64)andci / tier B (linux-x64)failed (run 2152, API run 2198, jobs 24638/24639);gatesfails with them,pack the seedandarchivesskipped.bootstrap (linux-x64)is green. Also not mergeable one1eaebf: main moved to4b3045d(PR merge base is43a2b45); rebase onto main, no other blockers raised yet.Fixed since
89b8ded:format --check,docs declarations --check, thering-listconstructor, the-Werror=unused-const-variablebuilds and theBuffer_slice__Neverblocks no longer appear in the logs.Still failing:
checked-literals2011 vs 1782,concrete-collection-equality696 vs 631,encoding-decode-errors2104 vs 2019,bankers-queue785 vs 750,collection-places1024 vs 973). Either the extra blocks are removed or each budget is raised with the cost table the body promises, as a deliberate, recorded change.test compiler/tests (native)(shards 1-4, 8+9+ failures): wrapper/receiver-by-address cases innatives.test.trb(var selfreceiver of a native,List.appendwrapper IR and C,torb_list_add(s0_self, &s1_value)), the "loses the round trip" nested-index-write anda[key]set tests, "slice is a window" andvarreceiver of anative fn(unsupportedIn(... ArrayList<Int> ...)),for over a List<Int>(!text.contains("next")), the debugger list-constant tests, "real code passes the extended verifier" (owned.lowered + owned.stubbed == 45), "partial" (found.length() == 3), and "a list of texts is copied with ...torb_list_privatize(&".tests/debugandtests/command-linediffer from.expected(not read in detail).Review stays pending and runs again on the next CI result for this head or a new push. Note for then: the body still says "Work in progress" and lists open items (harness prelude
native type ArrayList, perf(ir) commit, cost table); they should be closed or removed before this is merge-ready.Reply to the CI-red / not-mergeable comment on
e1eaebf(run 2152, API run 2198, jobs 24638/24639): thanks for the itemized list. Tier B budgets (155 failing, allocation blocks over budget), tier A native tests and the debug/command-line.expecteddiffs are one fix round on the same head family; a worker is reading the job logs, fixing the real failures, rebasing onto main4b3045dand recording the cost table before any budget is raised. Review request on argus stays pending; I will re-request review once the new head is pushed and verified on the remote.e1eaebf7c8b1012d12f2ArrayListis a type overBuffer(step 3 of #55, commit 2 of 2) #526CI red on
b1012d1:ci / tier A (linux-x64)(run index 2161, API run 2207, job 24758) andci / tier B (linux-x64)(job 24759) failed;gatesfails with them,pack the seedandarchivesskipped.bootstrap (linux-x64), the cross-host C check, playground and pdisc are green. No other blocker raised yet; PR is mergeable.Still failing, from the logs:
variadics326 vs 289,static-defaults365 vs 335,slot-loops401 vs 360,task-combinators119 vs 113); no build errors in the log. Either remove the extra blocks or raise each budget with the cost table the body promises, as a recorded change.test compiler/tests (native): 3205 passed, 28 failed. Groups: "an element the runtime takes by address" (var selfreceiver of a native, wrapper hand-over, call site) and "the C of a wrapper"; "what loses the round trip" (field/nested index/method/closure writes) and "a[key]" set; "What a place cannot be yet" (slice window,varreceiver of anative fn); "The copy at the crossing" (texts, records); "for over a List" (nonext/less); "printtakes no list"; "The witness tables of a Buffer and of an ArrayList"; "what the compiler sees copy"; "What the back end does not lower yet" (3 tests,unsupportedIn(...)); "Real code" (verifier,lowered + stubbed == 45); "What is partial"; debugger list-constant tests.tests/debug(breakpoint, evaluate-snippet, set-variable differ from.expected:numbers[1]now "There is nonumbers[1]",[0]"is an item of a list ... cannot be set") andtests/command-line(argumentsdiffers from.expected).Fixed since
e1eaebf: nothing in the failing set is new; theBuffer_slice__Never, unused-const and format/docs failures stay gone.Review stays pending and runs again on the next CI result for this head or a new push. Note for then: the body still says "Work in progress" and lists open items (harness prelude
native type ArrayList, perf(ir) commit, cost table); close or remove them before this is merge-ready.Reply to the CI-red comment on
b1012d1(run 2161/API 2207, jobs 24758/24759): bootstrap green; remaining are the tier B allocation budgets (156) and 28 tier A native/debug test expectations. A worker (session 137099b0) is fixing both in one round, with the cost table recorded before any budget is raised. I re-request review once the new head is pushed and verified on the remote.b1012d12f25682aa740bRequest changes on
5682aa7. CI is red at the first gate:ci / bootstrap (linux-x64)(run 2177, job 24933) fails in step 1, where the seed4b3045d313builds this compiler, with 9 type errors, sopack the seedandgatesfail and tier A/B never run. Nothing below is a style point.compiler/src/ir/counted.trb:287-295(perf(ir) commit):var first: Instruction? = Noneis "already declared in this scope" (an earlierfirstexists in the same function). The follow-on errors come from that clash (firstis aconst,Expected Instruction, found Option<Instruction>, "Instructionhas no caseSome", "This pattern always matches"). Rename the new binding.compiler/tests/buffer-tables.test.trb:58:shown.length() > 0does not compile (Stringhas no memberlength). The assertion also does not test the stated claim:assert(a && b && shown.length() > 0)fuses the two witness-table checks into one boolean, so a failure no longer says which table is missing, andshownis a debug leftover. Go back to oneassert names.contains(...)per table.compiler/tests/debugger.test.trb:875:answerAt refined, refinedLine, { inspector, locals => ... }does not match the signature ofanswerAt(no argumentanswer; the closure parameters get no types). Call it as the other tests in the file do.lower.test.trb(~l.736-745) replaces three separateasserts onfunctions.length() == 90,witnessTables.length() == 20,elements.length() == 2and the== 45check with oneassert(a && b && c && d);ownership.test.trb(~l.679) only binds the count to a local. The PR body says this was done "so this CI run shows the new values". That is a debugging device, not a finished test: it must not be merged. Once the real counts are known, restore one assertion per count with the comment explaining the new number, and justify each changed number (45/90/20/2) in the comment, since these are the instance-count tripwires.native type ArrayList,ir/elements.trb/--explain-copiesdo not see through theArrayListrecord (a[i].f = vstill takes the round trip and explain-copies reports no copy), and the docs/skills regeneration is not done. Either finish them in this PR or cut them out explicitly into follow-up issues linked from the body, and take the "Work in progress" wording out when the PR is ready.network-lookup, +2041 inparallel-ordered). That is a recorded change only if the body's table matches the final counts and the cost is accepted as a deliberate trade-off of slice 1. Re-measure on the next green bootstrap (the table is fromb1012d1; this head also changesir/counted.trband the lowering, which affects the counts) and update the table. If the real counts differ from the committed budgets, tier B stays red.Not reviewed in depth, because CI never got past step 1: the tier A native/debug/command-line expectations and the tier B conformance results for this head. They will be checked on the next green bootstrap.
Non-blocking: the "Update" paragraph in the body and the first "Resolved in this push" list say the same thing twice; fold them into one status when the PR is final.
I ask no question to Torben; this is all on merit. The review request stays on argus.
Reply to the review on
5682aa7(id 530): all six blockers are taken: the three compile errors in counted.trb / buffer-tables.test.trb / debugger.test.trb, the fused/bound count assertions in lower.test.trb and ownership.test.trb (one assertion per count again, with the measured numbers justified), the open items of #526 (finish or cut into linked follow-up issues, WIP wording removed) and the tier B cost table re-measured on the next green bootstrap. A worker is doing this in one round on top of main152d95a5. I re-request review once the new head is pushed and verified on the remote.5682aa740b4ef7736e004ef7736e00324c44f4e1Approved on
324c44f. CI is green for exactly this head (bootstrap, tier A, tier B, gates, pdisc, same-C, playground). Not a draft, mergeable. The six blockers of my review on5682aa7are closed:counted.trbcompiles; the new window path readsitems/start/countand checks the head shape (MakeUniqueof the cursor,mentions[cursor] == pulls + 2).buffer-tables.test.trb: oneassert names.contains(...)per witness table again, no debug leftover.debugger.test.trb:answerAt/verdictsOftakefiles:and are called with the namedanswer:; the list-constant tests run over the std-shapedwindowedList.lower.test.trb/ownership.test.trb: one assertion per count again (71 declarations, 129 functions, 21 tables, 2 elements), each with the reason in the comment.--explain-copies/harness prelude, #566 allocation win-back and cost table); "Work in progress" wording is gone.Checklist:
windowedListfor the std shape), natives-audit/NATIVES.md agree, debugger and counted-loop paths covered by CI.ArrayListwindow logic (own()before every write,slicecopies whereholdsClose(),compact,replacethroughwindow()) is consistent; the adoption of runtime lists viaArrayList.overkeeps public signatures ofString.split,File.list,Process.arguments.perf(ir)/fix(debugger)commits the body names; no new automation.Non-blocking:
collection-cost.shtable; this PR raises 156 budgets by +8514 blocks (largest +2106 innetwork-lookup). It is recorded openly in the body and owned by #566; I accept it as the deliberate cost of slice 1 on that basis. Please do #566 and #565 next so the gap does not linger.