Repository navigation
fix(engine): degrade when a helper thread cannot be created (#1122) - #2226
Merged
Merged
Conversation
Motivation: under memory pressure the node died with SIGABRT (exit 134). The captured panic was `thread 'xerj-drain-free' panicked: failed to allocate an alternative stack: Cannot allocate memory (os error 12)`. Release builds use panic = "abort", so that one panic is a whole-node outage. The drain 8000 -free path treated Builder::spawn's Err as the failure mode and dropped the generation inline, but std panics inside the child after pthread_create has already returned Ok. Root cause: on Unix, std::thread's thread_start calls stack_overflow::Handler::new, which mmaps an alternate signal stack and panics on MAP_FAILED. The parent cannot catch that panic, and panic = "abort" makes catch_unwind a no-op. The same family hits every fire-and-forget Builder::spawn and rayon's default worker spawn. Change: StdSpawn on Unix calls pthread_create with a 2 MiB stack and does not install a signal stack, so a create failure is io::Error and the job is handed back. Callers degrade and log once: drain-free drops the job on the caller, pool work runs inline, infinite workers (WAL fsync, merge) are skipped, diagnostics and share/ONNX init fail that step. Rayon pools in the engine use the same spawner. panic = "abort" is unchanged. The memory growth that reaches the allocation is not addressed. Evidence (rustc 1.97.1, from engine/, --locked): - cargo test -p xerj-common --lib thread_spawn: 10 passed, including an_enomem_setup_failure_returns_the_job_instead_of_panicking (os error 12, job handed back) and a_spawned_helper_does_not_install_an_alternate_signal_stack (SS_DISABLE on the helper, clear on a std::thread in the same process). - cargo test --lib -p xerj-common -p xerj-engine -p xerj-storage -p xerj-ai: 285, 797, 228, and 21 passed. cargo test -p xerj-server --bin xerj: 111 passed. 0 failed. - cargo test -p xerj-ai --features onnx-experimental --lib: 45 passed, 1 ignored. - cargo clippy -p xerj-common -p xerj-engine -p xerj-storage -p xerj-server -p xerj-ai --all-targets -- -D warnings: exit 0. - cargo clippy -p xerj-server --features debug-profiling --all-targets -- -D warnings: exit 0. - cargo clippy -p xerj-ai --features onnx-experimental --all-targets -- -D warnings: exit 0. - cargo fmt --all -- --check: exit 0. - cargo build --profile quick -p xerj-common: finished (panic = "abort" branch of the thread entry compiles). Assumed: Windows CreateThread failure stays an Err from Builder::spawn (no sigaltstack on that target). Not run on Windows. A global allocator abort (`memory allocation of N bytes failed`) is a different failure and is not handled here. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
Owner
Author
|
Xerj Merger made the fix: best-effort helper threads are now created through a non-panicking spawn (pthread_create on Unix) so a failed signal-stack allocation under memory pressure returns an error and the caller degrades (drop inline, run inline, retry, or fail that startup step) instead of aborting the node. Refs #1122, which stays open for the remaining memory-exhaustion paths. Diff reviewed; Build + Test green on this head. |
cursor Bot
pushed a commit
that referenced
this pull request
Oct 23, 2026
#2224, #2225, and #2226 landed on main after this branch. The only conflict was CHANGELOG.md: both Unreleased Fixed notes are kept, the #1947 semantic_text highlight entry and the #1122 best-effort thread entry. Engine files did not overlap (highlights stay in index.rs; helper-thread call sites are elsewhere) and merged cleanly. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
6 of 8 tasks
This was referenced Oct 23, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #1122 (partial fix; issue stays open)
What this changes, and why
Under memory pressure the node died with SIGABRT (exit 134). The captured panic was
thread 'xerj-drain-free' panicked: failed to allocate an alternative stack: Cannot allocate memory (os error 12).FrozenClaim::retirespawnedxerj-drain-freewithBuilder::spawnand treatedErras "drop the generation inline". On Unix,std::threadreturnsOkfrompthread_createand then, inside the child,stack_overflow::Handler::newmmaps an alternate signal stack and panics onENOMEM. The release profile ispanic = "abort", so that child panic is not catchable and kills the process.catch_unwinddoes not help: abort mode makes it a no-op, and the panic happens before the user's closure runs.StdSpawnon Unix now callspthread_createdirectly (2 MiB stack, the same default std uses) and does not install that signal stack. A create failure isio::Error, and the closure is handed back inSpawnFailure::job. Callers degrade and log once:spawnwhose pool could not start runs the finite job inlineThe governor watermark gate still refuses best-effort spawns (
try_spawn) once RSS is already high. Threads that must be attempted above the watermark usetry_spawn_unguardedand still getErrinstead of an abort.panic = "abort"is unchanged.What remains
The memory growth that reaches this allocation is out of scope (the ceiling tracked with #1032). So are the other abort shapes in the later comments: a Windows global-allocator abort (
memory allocation of N bytes failed, no pagefile) and the WSL2 silent VM death. An allocation inside the spawn path itself (the extraBoxaround the job) can still abort that way; this change only stops the signal-stack panic and a failedpthread_createfrom taking the node down.xerj autoindexstill starts workers withstd::thread::Builder. Several of those sites already handleErr, but the child signal-stack panic remains ifpthread_createsucceeds and the mmap fails. That is the CLI process, not the node, and it is left for a follow-up.Unix helper threads set the OS comm name (15 characters) and do not set
std::thread's name, so a panic on one of them is reported as<unnamed>.MergeExecutor::spawn_backgroundnow returnsSpawnedThreadinstead ofJoinHandle; nothing in this repo calls it.Evidence
Errand returns the jobthread_spawn::try_spawn_withan_enomem_setup_failure_returns_the_job_instead_of_panickingmmapfailurespawn_pthreada_spawned_helper_does_not_install_an_alternate_signal_stackpools::safe_rayon_spawnthe_safe_rayon_spawner_runs_pool_workpthread_createto fail in those cratesChecks
cargo fmt --all(CI runsrustfmt --checkandclippy -D warnings)cargo build --release -j 32 -p <crate>cargo test -p <crate>passes for the lib tests of the crates touched, andxerj-server --bin xerjengine/tests/es-compat-yaml/yaml/CHANGELOG.mdUnreleased)docs/CONTRIBUTION_REVIEW.md(optional fast path stays optional: inline drop or skip; no durable-state transition changed; feature-gated ONNX path compiled and tested; debug-profiling compiled with clippy, tests not executed)Not run:
cargo build --release(fat LTO).cargo build --profile quick -p xerj-commonwas run instead becausequickinheritspanic = "abort"and that is the branch the unit tests do not compile. ES-YAML was not run: the diff does not change query or wire behaviour. Windows was not run. debug-profiling unit tests were not executed;cargo clippy -p xerj-server --features debug-profiling --all-targets -- -D warningsexited 0. No workflow file was edited. debug-profiling is still absent from CI, which is pre-existing.Provenance
--locked.CreateThreadfailure is already anErrfromBuilder::spawn(there is no sigaltstack mmap on that target). A run on Windows where a failedCreateThreadstill aborts would falsify that. A global allocator abort is a different failure and is not handled.