Repository navigation
fix(engine): measure flush deadlines on the wall clock, not a paused tokio clock - #2167
Merged
Merged
Conversation
…tokio clock #2129 bounded the flush admission and flush-task waits with tokio::time::timeout. Under #[tokio::test(start_paused = true)] the runtime auto-advances to its next timer when idle, so a flush waiting on the process-global finalize gate (held by another test's runtime) hit the 120s deadline instantly and failed with 'timed out after 120s waiting for the finalize gate'. Measure both deadlines with std::time::Instant and, when the tokio timer fires early, wait a real slice on the blocking pool.
Owner
Author
|
Xerj Merger made the fix: flush deadlines are now measured on the wall clock ( Local CI on
On GitHub, every check that has finished passed; Build + Test was still queued. |
This was referenced Oct 22, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause
#2129 (Fixes #1635) bounded the flush admission waits (the flush semaphore and the process-global finalize gate) and the spawned flush-task wait with
tokio::time::timeout. That timer runs on tokio's clock, not the wall clock.The engine tests
a_background_merge_tick_cannot_move_a_hand_driven_fixture_segment_setandidle_index_schedules_no_merge_evaluations_without_segment_changesrun under#[tokio::test(start_paused = true)]. A paused runtime that goes idle auto-advances straight to its next timer.The finalize gate is shared across the whole process. At default test parallelism, another test (on another runtime) is often holding a permit. A paused-clock flush waiting for that permit is idle on its own runtime, so tokio jumped 120 s ahead at once and the flush failed with:
index [tick-guard] flush timed out after 120s waiting for the finalize gate; the memtable was not frozen, so this flush can be retriedThe flush itself was healthy. This is a regression from #2129, not timing noise:
Main
6c5f08f5: the fullxerj-enginelib suite at default parallelism failed 20/20 runs. Both tests pass when run alone.Bisect (5 runs each):
28de5cfc(before fix(engine): bound flush admission so refresh cannot hang (#1635) #2129): 0/5 failed1ec3ca57(fix(engine): bound flush admission so refresh cannot hang (#1635) #2129): 5/5 failed1c314d49(fix(engine): answer text exists without hydrating the corpus #2130): 5/5 failed533066b9(fix(autoindex): stop multi-array JSON files from wedging ingest #2131): 5/5 failedfix(engine): answer text exists without hydrating the corpus #2130, fix(autoindex): stop multi-array JSON files from wedging ingest #2131/docs(feedback): field report for multi-array JSON autoindex #2132 and fix(autoindex): a reduced plan no longer deletes an existing index #2128 are not involved.
Why the local runs missed it: my local runs used
--test-threads=2, which rarely overlaps the gate holder. CI's "default parallelism" step does.Production uses a real clock, so its deadlines were already correct. Only paused-clock runtimes were affected.
Fix
within_wall_clock, measures both deadlines (admission and flush task) withstd::time::Instant.tokio::time::timeout.The tests are not weakened. Three new paused-clock tests:
a_paused_clock_does_not_expire_a_flush_admission_wait: a permit released after 200 ms of real time is acquired within the 120 s budget. On main this fails instantly.a_paused_clock_still_enforces_the_wall_clock_flush_deadlines: the 100 ms admission deadline and the 100 ms flush-task deadline still fire on the wall clock (elapsed ≥ 100 ms). A flush task that really finishes in 150 ms passes.a_paused_clock_flush_waits_for_a_finalize_gate_held_elsewhere: the CI failure end to end. A paused-clock index flushes while another runtime holds every finalize-gate permit for 200 ms.Validation
cargo fmt --checkpasses.xerj-enginelib suite at default parallelism passed 30/30 runs. On main the same run failed 20/20.a_flush_that_never_finishes_fails_that_index_and_the_run_continuesran together 100 times with 0 failures.-D warningsandci-testworkspace tests): results will follow in a comment.This unblocks Build + Test on main and on #2141.