8000 fix(daemon): try again on each tick the shells reconcile couldn't settle at start by inkless · Pull Request #408 · inkless/bazhen · GitHub
[go: up one dir, main page]

Skip to content

fix(daemon): try again on each tick the shells reconcile couldn't settle at start - #408

Merged
inkless merged 3 commits into
mainfrom
gz/bz-388-retry-reconcile-on-the-tick-after-a-boot-failure
Nov 22, 2026
Merged

inkless merged 3 commits into
mainfrom
gz/bz-388-retry-reconcile-on-the-tick-after-a-boot-failure

Conversation

@inkless
@inkless inkless commented Nov 22, 2026 •
Copy link
Copy Markdown
Owner

What

At start, the Rust daemon's reconcile gave up for good when it couldn't reach the holders or read their inventory. That happens whenever a recorded holder has lost its socket: the health check tells it to bind again and answers unreachable, and the holder answers a tick later. Two things then stuck for the daemon's whole life:

  • a shell the dead daemon left creating held its start slot;
  • the running shells were never watched, so their exits and losses went unseen.

Now the start's pass keeps the ids of the shells it couldn't settle, and the 5 s tick tries those again, first in the tick and under its stop (shells/reconcile.rs, serve.rs::every_tick).

  • Only the start's shells are retried. A shell opened since is settled by its own open. start records creating before it takes the shell's turn, so a retry that re-read every recoverable shell could fail a shell mid-create as daemon_restart.

  • Each settle takes the shell's turn and re-reads it, at start too. A close, or an exit, loss or adopt 8000 ion event that went first, is respected. A shell found closed, lost or failed meanwhile is dropped untouched.

  • Kept or dropped. A shell stays kept only for a failure a later try may get past:

    • the store couldn't be read;
    • the runtime couldn't be reached;
    • the inventory couldn't be read;
    • a close or unlink failed in a way TerminalError::forceable() names;
    • the runtime doesn't list it, and what ran it isn't proven gone. A live holder with no socket whose rebind failed for now (a /proc read, or the kernel refusing the send) may yet bind again, or be shown gone. gone can't tell that from an outdated holder, so both are kept, said once as "may still run on its old server; trying again each tick".

    A definite answer drops it: listed, or provably gone. So does a failure no try can get past, said once.

  • Said once. Each pass collects its warnings and logs only those the last pass didn't say. A lasting failure is logged once, not every 5 s. An unreachable runtime now gets a warning; that return was silent. When the last kept shell settles, an info line says so.

  • No backoff. A retry costs what the start's pass did: one health read, one inventory, one turn per kept shell. The tick already reads the same holders' foregrounds and compatibility every 5 s. While an outdated holder runs, its shell stays kept, so the inventory runs each tick: one connect per holder. watches.adopt returns at once for a holder it already watches. Such a shell is now marked lost if its holder later dies, which nothing did before.

  • Stop. The retry runs under the tick's stop, which can end it amid a close or unlink. The shell is then left closing, or still naming its agent where it runs, and the next start's reconcile settles it. every_tick's doc says so.

A kept shell that never settles

If the runtime stays unreadable, the kept shells are retried every 5 s for as long as the daemon runs, logged once. This is intended: nothing is concluded about a runtime that can't be read.

  • A kept running or closing shell has a way out through the API. A close with confirmBusy records it closing, then fails on the runtime. A forced close then records it closed, and the next pass drops it.
  • A kept creating shell frees its start slot only when the runtime answers. No command is offered on a creating shell but rename. Offering a way to close one is tracked separately, since it's a change to the command surface, not to the retry.

Behaviour change: the start refuses when it can't read the shells it kept

If reconcile's read of the store fails at start, the daemon now refuses to start, where it used to warn and run with those shells unreconciled. With no ids read there's nothing to keep, and retrying a full read would race new opens, as above. The start already refuses on the same failure at its next read, the open terminals.

One bad row now blocks the start, so the refusal names it and says what to do. Once reconcile's read has failed, a second read, Reads::unreadable_shell, finds the first shell row that doesn't parse, by its id as stored:

bazhen can't read shell <id> kept from before: <SQLite's error>; start the bazhen build that wrote it, or move <data>/bazhen.db aside to start without anything it keeps

e.g. bazhen can't read shell terminal_… kept from before: Conversion error from type Text at index: 1, its state isn't one a shell has; start the bazhen build that wrote it, …. Where no row is to blame, such as an I/O error, it says bazhen can't read its shells: <SQLite's error>. The remedy leaves out editing the row by hand, since terminals is a projection of the event log.

Tested by serve::tests::refuses_to_start_when_it_cant_read_the_shells_kept_from_before. Controls:

  • start ignores reconcile's error -> refused None. The next read doesn't count a state it can't parse, so the start went on.
  • the refusal names no shell -> Some("bazhen can't read its shells: Conversion error from type Text at index: 1, its state isn't one a shell has").

Tests

Eleven new host tests on the in-memory terminals, none of which starts a holder or signals, and one sandbox test. Each has a control:

Test Broke Failed with
shells::reconcile::tests::settles_on_a_later_try_a_shell_left_creating_while_the_runtime_couldnt_be_reached unreachable arm keeps nothing ((Creating,1),(Creating,1)) vs ((Creating,1),(Failed,0))
shells::reconcile::tests::marks_lost_on_a_later_try_a_shell_whose_holder_went_while_the_runtime_couldnt_be_read inventory-error arm keeps nothing (Running,Running) vs (Running,Lost)
shells::reconcile::tests::never_settles_on_a_later_try_a_shell_opened_since_the_start retry reads every recoverable shell [Failed,Failed] vs [Failed,Creating]
shells::reconcile::tests::marks_lost_on_a_later_try_a_shell_whose_holder_wasnt_yet_shown_gone_at_start an unproven shell dropped (Running,Running) vs (Running,Lost)
shells::reconcile::tests::leaves_a_kept_shell_closed_while_a_later_try_read_the_runtime_as_it_was_closed settles the row read before the inventory, not the one re-read under the turn (Closed,4,1,0) vs (Closed,4,0,1)
shells::reconcile::tests::says_a_lasting_failure_once_and_once_that_the_shells_it_kept_are_settled says every warning each pass (3,1) vs (1,1)
shells::reconcile::tests::closes_on_a_later_try_an_exited_shell_whose_close_failed_for_now a retryable close failure taken as final (Closing,Closing) vs (Closing,Closed)
shells::reconcile::tests::tries_no_more_an_exited_shell_no_close_can_end a final close failure kept (Closing,2,0) vs (Closing,1,1)
serve::tests::tries_again_on_the_next_tick_a_shell_the_start_couldnt_settle every_tick doesn't retry Creating vs Failed
serve::tests::stops_at_once_amid_a_tick_trying_again_a_shell_the_start_couldnt_settle retry run outside the tick's stop every_tick didn't return in time
serve::tests::refuses_to_start_when_it_cant_read_the_shells_kept_from_before start ignores reconcile's error; the refusal names no shell None; the unnamed message

Sandbox test, run in moon run :test-real -- rust: shells::reconcile::tests::waits_for_a_users_close_of_a_kept_shell_rather_than_close_it_under_it. It pins the shell's turn in the retry. Without the turn:

  • a user's close records closing, and the holder ends the terminal;
  • a retry re-reads closing, finds it gone, and records closed itself;
  • the close then fails to record closed.

The test calls the real Shells::close, held in the fake's close once its terminal is gone. It runs the retry, then releases the close, and asserts the close answers with exactly one closed event. The retry is ordered by the turn's Arc count, which rises above two once a waiter queues behind the close, with a yield between reads. That count is the map's, the close's guard's and the waiter's. The wait panics after 10 s, so a regression fails the test rather than hanging it. It's a #[ignore] sandbox test because it calls Shells::close.

Run in moon run :test-real -- rust, narrowed to this test: it passes. Control: the turn in settle_now made a no-op -> (false, Some(Closed), 1) vs (true, Some(Closed), 1). The close fails with Internal("terminal.closed refused: terminal cannot handle close.completed while closed").

The tests are ordered by holds on the fake, not by sleeps or the writer's speed. The fake gains hold_inventory, hold_compatibility and hold_closes, all test-only. The close-race host test holds the retry at its inventory read, records a close meanwhile (as the system, with no idempotency answer) with the terminal gone, then lets the retry go. It proves the re-read; the sandbox test above proves the turn.

At start, every settle now takes the shell's turn. The existing reconcile tests run through that path unchanged, so they pin that the start's result is the same:

  • lost;
  • exited while down;
  • adoption, both cases;
  • creating, appeared or not, and appeared then exited;
  • closing finished;
  • unlink;
  • unreachable;
  • unreadable;
  • unproven;
  • another runtime.

holder.rs's health comment now says reconcile's retry also tells a holder to rebind, not only the start.

Not changed: the TypeScript daemon has the same gap. Its start order relies on the health check's rebind, it runs two runtimes, and its commands serialize through a queue rather than a per-shell turn, so it is tracked separately.

moon run :check passes on the head.

🤖 Generated with Claude Code

@inkless
inkless force-pushed the gz/bz-388-retry-reconcile-on-the-tick-after-a-boot-failure branch from 28a861b to a4db53e Compare November 22, 2026 22:31
inkless and others added 3 commits November 22, 2026 14:32
…tle at start

At start, reconcile gave up for good when the holders couldn't be reached
or their inventory couldn't be read. That happens whenever a recorded holder
has lost its socket and is told to bind it again, and it answers a tick later.
A shell the dead daemon left creating then held a start slot for the daemon's
whole life, and running shells were never watched.

The start's pass now keeps the ids of the shells it couldn't settle, and
each tick tries those again, first, under the tick's stop. Only those ids are
retried: a shell opened since is settled by its own open, and a retry that
re-read every recoverable shell could fail one mid-create. Each settle, at start
too, takes the shell's turn and re-reads it, so a close or an event that went
first is respected. Shells stay kept only for a failure a later try may get
past; a definite answer, or one no try can change, drops them. A lasting
failure is said once, and the pass that settles the last says so.

A start whose read of the kept shells fails now refuses, as it already did
at the read of the open terminals after it, rather than run with them
unreconciled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t read

Review fixes to the reconcile retry:

- A shell the runtime doesn't list, whose holder isn't proven gone, is
  kept and tried again each tick, said once, rather than dropped. A
  live holder with no socket whose rebind failed for now may yet bind,
  or be shown gone; `gone` can't tell it from an outdated holder, so
  that is kept too.
- The start's refusal names the shell whose row it can't read, and
  says to start the build that wrote it or move the database aside.
  The store's `unreadable_shell` finds it, read only once the start
  has failed to read the shells.
- A sandbox test pins that a retry waits for a user's close under the
  shell's turn. The fake's close can now be held once its terminal is
  gone.
- `every_tick`'s doc says a stop amid a retry's close or unlink may
  leave the shell for the next start to settle.
- The close-race test's comment no longer says its records are a
  user's.

Control: shells::reconcile::tests::marks_lost_on_a_later_try_a_shell_whose_holder_wasnt_yet_shown_gone_at_start: an unproven shell dropped -> left (Running,Running) right (Running,Lost)
Control: serve::tests::refuses_to_start_when_it_cant_read_the_shells_kept_from_before: the refusal names no shell -> Some("bazhen can't read its shells: Conversion error from type Text at index: 1, its state isn't one a shell has")

Controls of the retry itself, run on the commit before this one:

Control: shells::reconcile::tests::settles_on_a_later_try_a_shell_left_creating_while_the_runtime_couldnt_be_reached: unreachable arm keeps no shell -> left ((Creating,1),(Creating,1)) right ((Creating,1),(Failed,0))
Control: shells::reconcile::tests::marks_lost_on_a_later_try_a_shell_whose_holder_went_while_the_runtime_couldnt_be_read: inventory-error arm keeps no shell -> left (Running,Running) right (Running,Lost)
Control: shells::reconcile::tests::never_settles_on_a_later_try_a_shell_opened_since_the_start: retry reads recoverable_shells() instead of the kept ids -> left [Failed,Failed] right [Failed,Creating]
Control: shells::reconcile::tests::leaves_a_kept_shell_closed_while_a_later_try_read_the_runtime_as_it_was_closed: settles the row read before the inventory, not the one re-read under the turn -> left (Closed,4,1,0) right (Closed,4,0,1)
Control: serve::tests::refuses_to_start_when_it_cant_read_the_shells_kept_from_before: start ignores reconcile's error -> refused None
Control: shells::reconcile::tests::says_a_lasting_failure_once_and_once_that_the_shells_it_kept_are_settled: says every warning each pass -> left (3,1) right (1,1)
Control: shells::reconcile::tests::closes_on_a_later_try_an_exited_shell_whose_close_failed_for_now: Unended::Again mapped to Never -> left (Closing,Closing) right (Closing,Closed)
Control: shells::reconcile::tests::tries_no_more_an_exited_shell_no_close_can_end: Unended::Never mapped to Again -> left (Closing,2,0) right (Closing,1,1)
Control: serve::tests::tries_again_on_the_next_tick_a_shell_the_start_couldnt_settle: every_tick doesn't call reconcile_again -> left Creating right Failed
Control: serve::tests::stops_at_once_amid_a_tick_trying_again_a_shell_the_start_couldnt_settle: reconcile_again run outside stop.run_until_cancelled -> timeout, assert matches!(Ok(Ok(()))) failed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…und its turn test's wait

- The holder tests' reboot unwraps reconcile, which now says when it
  can't read the shells.
- The start-refusal test passes the start's new stop.
- The turn test's wait for a waiter on the shell's turn panics after
  10 s, so a retry that neither waits nor ends fails it rather than
  hanging.

Control: shells::reconcile::tests::waits_for_a_users_close_of_a_kept_shell_rather_than_close_it_under_it: the turn in settle_now made a no-op -> left (false, Some(Closed), 1) right (true, Some(Closed), 1), the close refused with Err(Internal("terminal.closed refused: terminal cannot handle close.completed while closed"))

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@inkless
inkless force-pushed the gz/bz-388-retry-reconcile-on-the-tick-after-a-boot-failure branch from a4db53e to c54b46a Compare November 22, 2026 22:41
@inkless
inkless merged commit f7ec489 into main Nov 22, 2026
28 checks passed
@inkless
inkless deleted the gz/bz-388-retry-reconcile-on-the-tick-after-a-boot-failure branch November 22, 2026 22:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0