Repository navigation
fix(daemon): try again on each tick the shells reconcile couldn't settle at start - #408
Merged
inkless merged 3 commits intoNov 22, 2026
Conversation
inkless
force-pushed
the
gz/bz-388-retry-reconcile-on-the-tick-after-a-boot-failure
branch
from
November 22, 2026 22:31
28a861b to
a4db53e
Compare
…tle at start At start, reconcile gave up for good when the holders couldn't be reached or their inventory couldn't be read. That happens whenever a recorded holder has lost its socket and is told to bind it again, and it answers a tick later. A shell the dead daemon left creating then held a start slot for the daemon's whole life, and running shells were never watched. The start's pass now keeps the ids of the shells it couldn't settle, and each tick tries those again, first, under the tick's stop. Only those ids are retried: a shell opened since is settled by its own open, and a retry that re-read every recoverable shell could fail one mid-create. Each settle, at start too, takes the shell's turn and re-reads it, so a close or an event that went first is respected. Shells stay kept only for a failure a later try may get past; a definite answer, or one no try can change, drops them. A lasting failure is said once, and the pass that settles the last says so. A start whose read of the kept shells fails now refuses, as it already did at the read of the open terminals after it, rather than run with them unreconciled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t read
Review fixes to the reconcile retry:
- A shell the runtime doesn't list, whose holder isn't proven gone, is
kept and tried again each tick, said once, rather than dropped. A
live holder with no socket whose rebind failed for now may yet bind,
or be shown gone; `gone` can't tell it from an outdated holder, so
that is kept too.
- The start's refusal names the shell whose row it can't read, and
says to start the build that wrote it or move the database aside.
The store's `unreadable_shell` finds it, read only once the start
has failed to read the shells.
- A sandbox test pins that a retry waits for a user's close under the
shell's turn. The fake's close can now be held once its terminal is
gone.
- `every_tick`'s doc says a stop amid a retry's close or unlink may
leave the shell for the next start to settle.
- The close-race test's comment no longer says its records are a
user's.
Control: shells::reconcile::tests::marks_lost_on_a_later_try_a_shell_whose_holder_wasnt_yet_shown_gone_at_start: an unproven shell dropped -> left (Running,Running) right (Running,Lost)
Control: serve::tests::refuses_to_start_when_it_cant_read_the_shells_kept_from_before: the refusal names no shell -> Some("bazhen can't read its shells: Conversion error from type Text at index: 1, its state isn't one a shell has")
Controls of the retry itself, run on the commit before this one:
Control: shells::reconcile::tests::settles_on_a_later_try_a_shell_left_creating_while_the_runtime_couldnt_be_reached: unreachable arm keeps no shell -> left ((Creating,1),(Creating,1)) right ((Creating,1),(Failed,0))
Control: shells::reconcile::tests::marks_lost_on_a_later_try_a_shell_whose_holder_went_while_the_runtime_couldnt_be_read: inventory-error arm keeps no shell -> left (Running,Running) right (Running,Lost)
Control: shells::reconcile::tests::never_settles_on_a_later_try_a_shell_opened_since_the_start: retry reads recoverable_shells() instead of the kept ids -> left [Failed,Failed] right [Failed,Creating]
Control: shells::reconcile::tests::leaves_a_kept_shell_closed_while_a_later_try_read_the_runtime_as_it_was_closed: settles the row read before the inventory, not the one re-read under the turn -> left (Closed,4,1,0) right (Closed,4,0,1)
Control: serve::tests::refuses_to_start_when_it_cant_read_the_shells_kept_from_before: start ignores reconcile's error -> refused None
Control: shells::reconcile::tests::says_a_lasting_failure_once_and_once_that_the_shells_it_kept_are_settled: says every warning each pass -> left (3,1) right (1,1)
Control: shells::reconcile::tests::closes_on_a_later_try_an_exited_shell_whose_close_failed_for_now: Unended::Again mapped to Never -> left (Closing,Closing) right (Closing,Closed)
Control: shells::reconcile::tests::tries_no_more_an_exited_shell_no_close_can_end: Unended::Never mapped to Again -> left (Closing,2,0) right (Closing,1,1)
Control: serve::tests::tries_again_on_the_next_tick_a_shell_the_start_couldnt_settle: every_tick doesn't call reconcile_again -> left Creating right Failed
Control: serve::tests::stops_at_once_amid_a_tick_trying_again_a_shell_the_start_couldnt_settle: reconcile_again run outside stop.run_until_cancelled -> timeout, assert matches!(Ok(Ok(()))) failed
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…und its turn test's wait
- The holder tests' reboot unwraps reconcile, which now says when it
can't read the shells.
- The start-refusal test passes the start's new stop.
- The turn test's wait for a waiter on the shell's turn panics after
10 s, so a retry that neither waits nor ends fails it rather than
hanging.
Control: shells::reconcile::tests::waits_for_a_users_close_of_a_kept_shell_rather_than_close_it_under_it: the turn in settle_now made a no-op -> left (false, Some(Closed), 1) right (true, Some(Closed), 1), the close refused with Err(Internal("terminal.closed refused: terminal cannot handle close.completed while closed"))
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
inkless
force-pushed
the
gz/bz-388-retry-reconcile-on-the-tick-after-a-boot-failure
branch
from
November 22, 2026 22:41
a4db53e to
c54b46a
Compare
inkless
deleted the
gz/bz-388-retry-reconcile-on-the-tick-after-a-boot-failure
branch
November 22, 2026 22:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
At start, the Rust daemon's reconcile gave up for good when it couldn't reach the holders or read their inventory. That happens whenever a recorded holder has lost its socket: the health check tells it to bind again and answers unreachable, and the holder answers a tick later. Two things then stuck for the daemon's whole life:
creatingheld its start slot;Now the start's pass keeps the ids of the shells it couldn't settle, and the 5 s tick tries those again, first in the tick and under its stop (
shells/reconcile.rs,serve.rs::every_tick).Only the start's shells are retried. A shell opened since is settled by its own open.
startrecordscreatingbefore it takes the shell's turn, so a retry that re-read every recoverable shell could fail a shell mid-create asdaemon_restart.Each settle takes the shell's turn and re-reads it, at start too. A close, or an exit, loss or adopt 8000 ion event that went first, is respected. A shell found closed, lost or failed meanwhile is dropped untouched.
Kept or dropped. A shell stays kept only for a failure a later try may get past:
TerminalError::forceable()names;/procread, or the kernel refusing the send) may yet bind again, or be shown gone.gonecan't tell that from an outdated holder, so both are kept, said once as "may still run on its old server; trying again each tick".A definite answer drops it: listed, or provably gone. So does a failure no try can get past, said once.
Said once. Each pass collects its warnings and logs only those the last pass didn't say. A lasting failure is logged once, not every 5 s. An unreachable runtime now gets a warning; that return was silent. When the last kept shell settles, an info line says so.
No backoff. A retry costs what the start's pass did: one health read, one inventory, one turn per kept shell. The tick already reads the same holders' foregrounds and compatibility every 5 s. While an outdated holder runs, its shell stays kept, so the inventory runs each tick: one connect per holder.
watches.adoptreturns at once for a holder it already watches. Such a shell is now marked lost if its holder later dies, which nothing did before.Stop. The retry runs under the tick's stop, which can end it amid a close or unlink. The shell is then left closing, or still naming its agent where it runs, and the next start's reconcile settles it.
every_tick's doc says so.A kept shell that never settles
If the runtime stays unreadable, the kept shells are retried every 5 s for as long as the daemon runs, logged once. This is intended: nothing is concluded about a runtime that can't be read.
confirmBusyrecords it closing, then fails on the runtime. A forced close then records it closed, and the next pass drops it.Behaviour change: the start refuses when it can't read the shells it kept
If reconcile's read of the store fails at start, the daemon now refuses to start, where it used to warn and run with those shells unreconciled. With no ids read there's nothing to keep, and retrying a full read would race new opens, as above. The start already refuses on the same failure at its next read, the open terminals.
One bad row now blocks the start, so the refusal names it and says what to do. Once reconcile's read has failed, a second read,
Reads::unreadable_shell, finds the first shell row that doesn't parse, by its id as stored:e.g.
bazhen can't read shell terminal_… kept from before: Conversion error from type Text at index: 1, its state isn't one a shell has; start the bazhen build that wrote it, …. Where no row is to blame, such as an I/O error, it saysbazhen can't read its shells: <SQLite's error>. The remedy leaves out editing the row by hand, sinceterminalsis a projection of the event log.Tested by
serve::tests::refuses_to_start_when_it_cant_read_the_shells_kept_from_before. Controls:None. The next read doesn't count a state it can't parse, so the start went on.Some("bazhen can't read its shells: Conversion error from type Text at index: 1, its state isn't one a shell has").Tests
Eleven new host tests on the in-memory terminals, none of which starts a holder or signals, and one sandbox test. Each has a control:
shells::reconcile::tests::settles_on_a_later_try_a_shell_left_creating_while_the_runtime_couldnt_be_reached((Creating,1),(Creating,1))vs((Creating,1),(Failed,0))shells::reconcile::tests::marks_lost_on_a_later_try_a_shell_whose_holder_went_while_the_runtime_couldnt_be_read(Running,Running)vs(Running,Lost)shells::reconcile::tests::never_settles_on_a_later_try_a_shell_opened_since_the_start[Failed,Failed]vs[Failed,Creating]shells::reconcile::tests::marks_lost_on_a_later_try_a_shell_whose_holder_wasnt_yet_shown_gone_at_start(Running,Running)vs(Running,Lost)shells::reconcile::tests::leaves_a_kept_shell_closed_while_a_later_try_read_the_runtime_as_it_was_closed(Closed,4,1,0)vs(Closed,4,0,1)shells::reconcile::tests::says_a_lasting_failure_once_and_once_that_the_shells_it_kept_are_settled(3,1)vs(1,1)shells::reconcile::tests::closes_on_a_later_try_an_exited_shell_whose_close_failed_for_now(Closing,Closing)vs(Closing,Closed)shells::reconcile::tests::tries_no_more_an_exited_shell_no_close_can_end(Closing,2,0)vs(Closing,1,1)serve::tests::tries_again_on_the_next_tick_a_shell_the_start_couldnt_settleevery_tickdoesn't retryCreatingvsFailedserve::tests::stops_at_once_amid_a_tick_trying_again_a_shell_the_start_couldnt_settleevery_tickdidn't return in timeserve::tests::refuses_to_start_when_it_cant_read_the_shells_kept_from_beforeNone; the unnamed messageSandbox test, run in
moon run :test-real -- rust:shells::reconcile::tests::waits_for_a_users_close_of_a_kept_shell_rather_than_close_it_under_it. It pins the shell's turn in the retry. Without the turn:closing, and the holder ends the terminal;closing, finds it gone, and recordscloseditself;closed.The test calls the real
Shells::close, held in the fake's close once its terminal is gone. It runs the retry, then releases the close, and asserts the close answers with exactly oneclosedevent. The retry is ordered by the turn'sArccount, which rises above two once a waiter queues behind the close, with a yield between reads. That count is the map's, the close's guard's and the waiter's. The wait panics after 10 s, so a regression fails the test rather than hanging it. It's a#[ignore]sandbox test because it callsShells::close.Run in
moon run :test-real -- rust, narrowed to this test: it passes. Control: the turn insettle_nowmade a no-op ->(false, Some(Closed), 1)vs(true, Some(Closed), 1). The close fails withInternal("terminal.closed refused: terminal cannot handle close.completed while closed").The tests are ordered by holds on the fake, not by sleeps or the writer's speed. The fake gains
hold_inventory,hold_compatibilityandhold_closes, all test-only. The close-race host test holds the retry at its inventory read, records a close meanwhile (as the system, with no idempotency answer) with the terminal gone, then lets the retry go. It proves the re-read; the sandbox test above proves the turn.At start, every settle now takes the shell's turn. The existing reconcile tests run through that path unchanged, so they pin that the start's result is the same:
holder.rs'shealthcomment now says reconcile's retry also tells a holder to rebind, not only the start.Not changed: the TypeScript daemon has the same gap. Its start order relies on the health check's rebind, it runs two runtimes, and its commands serialize through a queue rather than a per-shell turn, so it is tracked separately.
moon run :checkpasses on the head.🤖 Generated with Claude Code