8000 fix(engine): degrade when a helper thread cannot be created (#1122) by xerj-org · Pull Request #2226 · xerj-org/xerj · GitHub
[go: up one dir, main page]

Skip to content

fix(engine): degrade when a helper thread cannot be created (#1122) - #2226

Merged
xerj-org merged 1 commit into
mainfrom
cursor/spawn-setup-failure-degrade-a46d
Oct 23, 2026
Merged

xerj-org merged 1 commit into
mainfrom
cursor/spawn-setup-failure-degrade-a46d

Conversation

@xerj-org
@xerj-org xerj-org commented Oct 23, 2026 •
Copy link
Copy Markdown
Owner

Refs #1122 (partial fix; issue stays open)

What this changes, and why

Under memory pressure the node died with SIGABRT (exit 134). The captured panic was thread 'xerj-drain-free' panicked: failed to allocate an alternative stack: Cannot allocate memory (os error 12).

FrozenClaim::retire spawned xerj-drain-free with Builder::spawn and treated Err as "drop the generation inline". On Unix, std::thread returns Ok from pthread_create and then, inside the child, stack_overflow::Handler::new mmaps an alternate signal stack and panics on ENOMEM. The release profile is panic = "abort", so that child panic is not catchable and kills the process. catch_unwind does not help: abort mode makes it a no-op, and the panic happens before the user's closure runs.

StdSpawn on Unix now calls pthread_create directly (2 MiB stack, the same default std uses) and does not install that signal stack. A create failure is io::Error, and the closure is handed back in SpawnFailure::job. Callers degrade and log once:

  • drain-free drops the job on the caller, which frees the generation Arcs
  • engine rayon pools (ingest, background, merge, search) spawn workers the same way; a real create failure fails the pool build and the existing one-thread / inline fallback runs
  • a pool spawn whose pool could not start runs the finite job inline
  • WAL fsync and background merge skip the thread (the loops do not return) and retry on the next attempt; flush / rotate / shutdown still fsync
  • ingest-memory trace stays off; share-tunnel open fails over to the local link; ONNX init and debug-profile startup return an error

The governor watermark gate still refuses best-effort spawns (try_spawn) once RSS is already high. Threads that must be attempted above the watermark use try_spawn_unguarded and still get Err instead of an abort. panic = "abort" is unchanged.

What remains

The memory growth that reaches this allocation is out of scope (the ceiling tracked with #1032). So are the other abort shapes in the later comments: a Windows global-allocator abort (memory allocation of N bytes failed, no pagefile) and the WSL2 silent VM death. An allocation inside the spawn path itself (the extra Box around the job) can still abort that way; this change only stops the signal-stack panic and a failed pthread_create from taking the node down.

xerj autoindex still starts workers with std::thread::Builder. Several of those sites already handle Err, but the child signal-stack panic remains if pthread_create succeeds and the mmap fails. That is the CLI process, not the node, and it is left for a follow-up.

Unix helper threads set the OS comm name (15 characters) and do not set std::thread's name, so a panic on one of them is reported as <unnamed>. MergeExecutor::spawn_background now returns SpawnedThread instead of JoinHandle; nothing in this repo calls it.

Evidence

rustc 1.97.1, from engine/, cargo --locked

cargo test -p xerj-common --lib thread_spawn
  10 passed, 0 failed
  includes an_enomem_setup_failure_returns_the_job_instead_of_panicking
  (os error 12, job handed back and runnable inline)
  and a_spawned_helper_does_not_install_an_alternate_signal_stack
  (SS_DISABLE set on the helper; clear on a std::thread in the same process)

cargo test --lib -p xerj-common -p xerj-engine -p xerj-storage -p xerj-ai
  xerj-common 285 passed
  xerj-engine 797 passed
  xerj-storage 228 passed
  xerj-ai 21 passed
  0 failed

cargo test -p xerj-server --bin xerj
  111 passed, 0 failed

cargo test -p xerj-ai --features onnx-experimental --lib
  45 passed, 1 ignored, 0 failed
  (loader_panic test still passes; the synthetic panic prints as <unnamed>)

cargo clippy -p xerj-common -p xerj-engine -p xerj-storage -p xerj-server -p xerj-ai --all-targets -- -D warnings
  exit 0

cargo clippy -p xerj-server --features debug-profiling --all-targets -- -D warnings
  exit 0

cargo clippy -p xerj-ai --features onnx-experimental --all-targets -- -D warnings
  exit 0

cargo fmt --all -- --check
  exit 0

cargo build --profile quick -j 32 -p xerj-common
  finished (this profile inherits panic = "abort", so the non-unwind thread entry compiles)
Claim Source Test Result Gap
ENOMEM setup failure is Err and returns the job thread_spawn::try_spawn_with an_enomem_setup_failure_returns_the_job_instead_of_panicking pass injected spawner, not a live mmap failure
Helper does not install an alt stack; std threads in this process do spawn_pthread a_spawned_helper_does_not_install_an_alternate_signal_stack pass Linux only
Rayon pool work runs through the safe spawner pools::safe_rayon_spawn the_safe_rayon_spawner_runs_pool_work pass (inside the 797) does not exhaust memory
Drain-free / WAL / merge degrade without running an infinite loop inline call sites existing crate tests still pass pass no test forces pthread_create to fail in those crates

Checks

  • cargo fmt --all (CI runs rustfmt --check and clippy -D warnings)
  • Scoped release build of the crates touched: cargo build --release -j 32 -p <crate>
  • cargo test -p <crate> passes for the lib tests of the crates touched, and xerj-server --bin xerj
  • ES-YAML conformance suite at 0 failed
  • New ES-compatible behaviour has a matching YAML case under engine/tests/es-compat-yaml/yaml/
  • Docs updated if user-visible behaviour changed (CHANGELOG.md Unreleased)
  • For non-trivial changes: the applicable audit in docs/CONTRIBUTION_REVIEW.md (optional fast path stays optional: inline drop or skip; no durable-state transition changed; feature-gated ONNX path compiled and tested; debug-profiling compiled with clippy, tests not executed)
  • Operational-recipe audit: N/A, this PR does not change an operational document or runnable recipe

Not run: cargo build --release (fat LTO). cargo build --profile quick -p xerj-common was run instead because quick inherits panic = "abort" and that is the branch the unit tests do not compile. ES-YAML was not run: the diff does not change query or wire behaviour. Windows was not run. debug-profiling unit tests were not executed; cargo clippy -p xerj-server --features debug-profiling --all-targets -- -D warnings exited 0. No workflow file was edited. debug-profiling is still absent from CI, which is pre-existing.

Provenance

  • Written by: AI agent (Cursor cloud agent). The account that opens this pull request is the reviewer of record.
  • Verified (commands run, output observed): the commands and counts in Evidence, on rustc 1.97.1, --locked.
  • Assumed (not tested, and what would falsify it): Windows CreateThread failure is already an Err from Builder::spawn (there is no sigaltstack mmap on that target). A run on Windows where a failed CreateThread still aborts would falsify that. A global allocator abort is a different failure and is not handled.
Open in Web Open in Cursor 

Motivation: under memory pressure the node died with SIGABRT (exit 134).
The captured panic was `thread 'xerj-drain-free' panicked: failed to
allocate an alternative stack: Cannot allocate memory (os error 12)`.
Release builds use panic = "abort", so that one panic is a whole-node
outage. The drain
8000
-free path treated Builder::spawn's Err as the failure
mode and dropped the generation inline, but std panics inside the child
after pthread_create has already returned Ok.

Root cause: on Unix, std::thread's thread_start calls
stack_overflow::Handler::new, which mmaps an alternate signal stack and
panics on MAP_FAILED. The parent cannot catch that panic, and
panic = "abort" makes catch_unwind a no-op. The same family hits every
fire-and-forget Builder::spawn and rayon's default worker spawn.

Change: StdSpawn on Unix calls pthread_create with a 2 MiB stack and
does not install a signal stack, so a create failure is io::Error and
the job is handed back. Callers degrade and log once: drain-free drops
the job on the caller, pool work runs inline, infinite workers (WAL
fsync, merge) are skipped, diagnostics and share/ONNX init fail that
step. Rayon pools in the engine use the same spawner. panic = "abort"
is unchanged. The memory growth that reaches the allocation is not
addressed.

Evidence (rustc 1.97.1, from engine/, --locked):
- cargo test -p xerj-common --lib thread_spawn: 10 passed, including
  an_enomem_setup_failure_returns_the_job_instead_of_panicking (os
  error 12, job handed back) and
  a_spawned_helper_does_not_install_an_alternate_signal_stack
  (SS_DISABLE on the helper, clear on a std::thread in the same process).
- cargo test --lib -p xerj-common -p xerj-engine -p xerj-storage -p xerj-ai:
  285, 797, 228, and 21 passed. cargo test -p xerj-server --bin xerj:
  111 passed. 0 failed.
- cargo test -p xerj-ai --features onnx-experimental --lib: 45 passed,
  1 ignored.
- cargo clippy -p xerj-common -p xerj-engine -p xerj-storage -p xerj-server
  -p xerj-ai --all-targets -- -D warnings: exit 0.
- cargo clippy -p xerj-server --features debug-profiling --all-targets
  -- -D warnings: exit 0.
- cargo clippy -p xerj-ai --features onnx-experimental --all-targets
  -- -D warnings: exit 0.
- cargo fmt --all -- --check: exit 0.
- cargo build --profile quick -p xerj-common: finished (panic = "abort"
  branch of the thread entry compiles).

Assumed: Windows CreateThread failure stays an Err from Builder::spawn
(no sigaltstack on that target). Not run on Windows. A global allocator
abort (`memory allocation of N bytes failed`) is a different failure
and is not handled here.

Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
@cla-bot cla-bot Bot added the cla-signed label Oct 23, 2026
@xerj-org
Copy link
Copy Markdown
Owner Author

Xerj Merger made the fix: best-effort helper threads are now created through a non-panicking spawn (pthread_create on Unix) so a failed signal-stack allocation under memory pressure returns an error and the caller degrades (drop inline, run inline, retry, or fail that startup step) instead of aborting the node. Refs #1122, which stays open for the remaining memory-exhaustion paths. Diff reviewed; Build + Test green on this head.

@xerj-org
xerj-org merged commit 3a0fd82 into main Oct 23, 2026
24 checks passed
cursor Bot pushed a commit that referenced this pull request Oct 23, 2026
#2224, #2225, and #2226 landed on main after this branch. The only
conflict was CHANGELOG.md: both Unreleased Fixed notes are kept, the
#1947 semantic_text highlight entry and the #1122 best-effort thread
entry. Engine files did not overlap (highlights stay in index.rs;
helper-thread call sites are elsewhere) and merged cleanly.

Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0