Repository navigation
fix(engine): size the Windows memory cap from commit space (#1122) - #2236
Merged
Merged
Conversation
Motivation: on Windows 11 with 16 GiB RAM and no pagefile, a default `xerj autoindex` aborted the node with `memory allocation of N bytes failed` (field report #2174, rc.98). Commit limit is about physical RAM, and other processes had already charged most of it. The release profile is panic = "abort", so the global allocator abort takes the process down. PR #2226 stopped the Unix signal-stack spawn panic and left this shape open. Root cause: the AUTO tier is chosen from physical RAM. The Windows probe added in #1766 reads GlobalMemoryStatusEx, including ullTotalPageFile (commit limit) and ullAvailPageFile (bytes this process can still commit), but only ullTotalPhys feeds the tier. A 16 GiB box therefore gets an 8 GiB process cap and a ~7.6 GiB RSS watermark while only ~4 GiB can still be committed. The breaker never engages. Working set can also sit under commit charge once the memory manager trims it, and with no pagefile the next VirtualAlloc fails. Change, Windows only: after the physical-RAM tier is chosen, lower the cap to process_commit + available commit, minus a reserve (one quarter of that ceiling, clamped to 512 MiB..2 GiB). The 64/128 GiB tier steps stay on physical RAM, so a pagefile that covers RAM does not move them, and a small kernel gap under RAM does not either. The sampler compares max(working set, commit charge) to the watermark and engages the breaker when available commit falls under the reserve. The 429 names the commit limit and how to enlarge the pagefile. Raising indices.breaker.total.limit cannot grant commit Windows will not give. panic = "abort" is unchanged. Linux and macOS take the None path and keep the previous cap, watermark, and 429 sentence. Evidence (rustc 1.97.1, from engine/, --locked): - cargo test -p xerj-common --lib: 290 passed, 0 failed. Includes a_pagefileless_16gib_box_is_capped_by_available_commit (16 GiB physical, commit limit 16 GiB, 4 GiB available, 200 MiB already charged → effective 3378511872 bytes, under the 8 GiB tier, watermark headroom ≥ 512 MiB). - cargo test -p xerj-engine --lib governor::: 38 passed, 0 failed. Includes the tier staying at 8 GiB for that sample, the exact pre-change 429 sentence when commit is not the ceiling, and commit_starvation_rejects_writes_and_releases_when_commit_returns (HTTP 429 while RSS is under the watermark). - cargo clippy -p xerj-common -p xerj-engine --all-targets -- -D warnings: exit 0. - cargo fmt --all -- --check: exit 0. Assumed: not run on Windows. A GlobalMemoryStatusEx whose ullTotalPageFile / ullAvailPageFile do not mean commit limit and remaining commit would falsify the ceiling. An allocation larger than the reserve (at most 2 GiB) can still abort between samples. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
The only conflict was CHANGELOG.md: both sides added an Unreleased Fixed entry at the same place. Keep both — the Windows commit-space memory cap (#1122) and main's semantic_text highlight fragments (#1947). Engine and common sources merged cleanly. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
5 of 8 tasks
The only conflict was CHANGELOG.md. Both sides added an Unreleased Fixed entry at the top of the section. Keep both: main's corpus-index journal resume (#2239) and this branch's Windows commit-space memory cap (#1122). The prefix-resolution Fixed entry from #2235 merged without a marker. Engine and common sources were not touched on main since the last merge and stayed as this branch left them. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
The only conflict was CHANGELOG.md. Both sides added an Unreleased Fixed entry in the same place. Keep both: main's `_validate/query` mapping check (#2263) and this branch's Windows commit-space memory cap (#1122). The AVX2 baseline note from #2253 merged without a marker and stays above them. engine.rs auto-merged. The cluster metadata proposer from #2257 (create_index_exact, delete_index_local, attach_cluster) is intact, and the sampler still calls current_process_memory_bytes plus refresh_memory_observing so the Windows commit ceiling stays in effect. xerj-common and governor.rs were not touched on main and are unchanged from this branch. No .github file was edited. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
The only content conflict was the parent RSS 429 in engine/crates/xerj-engine/src/governor.rs. Main (#2238) rewrote that sentence to name limits.max_process_memory_mb / XERJ_MAX_PROCESS_MEMORY_MB and still point at a live indices.breaker.total.limit. This branch (#1122) had moved the same sentence into memory_breaker_reason so a Windows commit ceiling can replace the hint with the commit limit and the pagefile. Keep the helper. The None arm now uses main's sentence, so an RSS rejection that is not commit-capped still names the process-memory cap. A commit ceiling still names the Windows commit limit and how to enlarge the pagefile, and does not suggest raising indices.breaker.total.limit. On a host where startup already lowered the cap, the hysteresis test accepts that commit sentence instead of failing the process-cap check. engine.rs auto-merged: the sampler still observes commit charge (#1122) and still requests a drain from ingest_pressure_needs_drain (#2238). CHANGELOG keeps both Unreleased notes. No .github file was edited. process_memory_matches_rss_off_windows compared two live RSS probes. Under --test-threads=8 those reads differed by 64–128 KiB, because the probe and sibling tests allocate between the two calls. It now requires one quiet pair in 128 tries. Nine governor-suite runs at 8 threads passed after that. Evidence (rustc 1.97.1, from engine/): - cargo test -p xerj-common --lib: 304 passed, 0 failed. - cargo test -p xerj-engine --lib governor::: 39 passed, 0 failed, then 9 more runs at --test-threads=8, all 39 passed. - cargo test -p xerj-engine --test memtable_budget_drain_unblocks_ingest: 1 passed, 0 failed. - cargo build -p xerj-engine: finished (dev profile). - cargo clippy -p xerj-common -p xerj-engine --all-targets -- -D warnings: exit 0, including the lib, its unit tests, the integration tests, and search_bench. - cargo fmt --all -- --check: exit 0. An agent wrote this merge. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
The use-case harness on c763939 failed in phase 5 (brain --fresh after a wiped data dir) with `rm: Directory not empty` on ax-journal and autoindex-catalog, then HTTP 500 `Segment … not found` while reading the catalog back. That job merged this branch into c16fad5, which does not contain the open-time repair. This is the same torn-snapshot failure #2244 hit and #2256 fixed on main (`drop_local_segments_missing_files`, `reopen_drops_a_snapshot_segment_whose_file_is_gone`). The Windows commit-budget change does not run on that Linux job: Linux and macOS still take the previous cap. No conflict. CHANGELOG keeps both Unreleased notes. No .github file was edited. An agent wrote this merge. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
Owner
Author
|
Xerj Merger made the fix: on Windows the AUTO memory cap is now lowered to the commit space actually available (minus a reserve), and the sampler compares max(working set, commit charge) to the watermark, so the breaker engages before VirtualAlloc fails. Partial fix for #1122, which stays open. |
cursor Bot
pushed a commit
that referenced
this pull request
Oct 23, 2026
Conflicts were governor.rs only. CHANGELOG.md auto-merged. engine.rs auto-merged, then the sampler called sample_drain_due with `rss` after main renamed that sample to `observed`. Both behaviours stay. #1981 (this branch): a flat footprint at or above the watermark releases the breaker after 2 seconds and does not re-arm inside the 4 MiB band. While that latch stays engaged, drain requests double from 2 seconds up to 15 minutes unless the sampled footprint falls by more than 4 MiB. corpus index reports eta_quality=stalled at 0/N and after 10 minutes of silence on a phase with no denominator. #2236 / #1122 (main): on Windows the process cap is limited by commit space. The sampler compares max(working set, commit charge) to the watermark, and available commit under the reserve engages the breaker even when that figure is under the watermark. The flat-RSS release does not clear a commit-starvation hold. The flat clock keeps running, so the first sample after commit recovers releases a footprint that has already been flat for 2 seconds. The drain schedule observes the same figure the breaker compared. The two drain clocks are unchanged: that sampled footprint uses MemoryDrainSchedule, and the summed-memtable latch stays on the fixed 2 second window. CHANGELOG keeps the #1981 entry and main's #1122 commit-space entry. #2290 did not edit CHANGELOG.md. No .github file was edited. Verified (rustc 1.97.1 (8bab26f4f 2026-07-14), from engine/): - cargo fmt --all --check: exit 0 - cargo clippy -p xerj-engine --all-targets -- -D warnings: exit 0 - cargo test -p xerj-engine --lib -- governor:: pressure_drain: 45 passed, 0 failed, 0.15s. Includes the #1981 flat-RSS and backoff tests, the #1122 commit-starvation test, and commit_starvation_holds_a_flat_footprint_until_commit_returns. - cargo test -p xerj-engine --test memtable_budget_drain_unblocks_ingest --test drain_releases_memory: 5 passed, 0 failed (4 + 1). - cargo test -p xerj-autoindex --lib progress::tests:: : 37 passed, 0 failed, 0.41s, including a_phase_at_zero_reports_a_stall_once_nothing_completes. Assumed: not run on Windows. The commit-starvation cases inject the shortage; process_memory_matches_rss_off_windows passed on this Linux host. Workspace-wide clippy and the ES-YAML gate were not run locally. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
This was referenced Oct 23, 2026
Merged
cursor Bot
pushed a commit
that referenced
this pull request
Oct 24, 2026
The only conflict was CHANGELOG.md. Both Unreleased fixes stay: the large-JSON ingest bound (#1122) and _refresh publishing an unflushed memtable without waiting on a background merge (#2234, #2245). #2236's Windows commit-space cap in governor.rs was already on this branch and was not rewritten. index.rs and xerj-autoindex kept the analyze reservation alongside the per-index refresh_once path. cargo +1.97.1 fmt --all made 8000 no further edits. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
cursor Bot
pushed a commit
that referenced
this pull request
Oct 24, 2026
main moved through 91fc16e, including per-index autoindex finalize refresh (#2245 / #2234), the Windows commit-space memory cap (#2236), and the plain-text flood-stage write block (#2290). Conflicts: - CHANGELOG.md. Both Unreleased Fixed notes stay: _refresh no longer waits on a background merge (#2234), and the neural ingest estimate plus the fixed-at-ingest --embed-mode wording (#1612). - engine/crates/xerj-autoindex/src/lib.rs. Both modules stay: memory_cli from main and neural_cost from this branch. Auto-merged engine, API, server, and llms-full.txt edits keep main's refresh, cluster, and hub behaviour. The ingest probe, embedder warnings, and fixed-at-ingest wording stay with them. Create-index and put-mapping still attach the warnings array on the cluster and standalone success paths, and the key stays omitted when the list is empty. Finalize still refreshes each dataset index on its own. Fixes #1612. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
10 of 13 tasks
Merged
9 tasks done
timothybrush
pushed a commit
to timothybrush/xerj
that referenced
this pull request
Oct 24, 2026
The merge of main inserted the xerj-org#2236 changelog bullet beside the release notes' own entry. The inserted bullet cited open issue xerj-org#1122 without saying the issue stays open, which the notes gate rejects. One entry remains: the Windows commit-space cap is partial, and xerj-org#1122 stays open. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #1122 (partial fix; issue stays open)
What this changes, and why
On Windows 11 with 16 GiB RAM and no pagefile, a default
xerj autoindexaborted the node withmemory allocation of N bytes failed(field report #2174, rc.98). The commit limit is about physical RAM, and other processes had already charged most of it. The release profile ispanic = "abort", so that allocator failure takes the process down. PR #2226 stopped the Unix signal-stack spawn panic and left this shape open.The AUTO tier is chosen from physical RAM. The Windows probe added in #1766 already calls
GlobalMemoryStatusEx, which fillsullTotalPageFile(commit limit) andullAvailPageFile(bytes this process can still commit), but onlyullTotalPhyswas used. A 16 GiB box therefore got an 8 GiB process cap and a ~7.6 GiB RSS watermark while only ~4 GiB could still be committed. The breaker never engaged. Working set can also sit under commit charge after the memory manager trims it, and with no pagefile the nextVirtualAllocfails.On Windows, after the physical-RAM tier is chosen, the cap is lowered to
process commit + available commit, minus a reserve (one quarter of that ceiling, clamped to 512 MiB..2 GiB). The 64 GiB / 128 GiB tier steps stay on physical RAM, so a pagefile that covers RAM does not move them, and a small kernel gap under RAM does not either. The sampler comparesmax(working set, commit charge)with the watermark, and engages the breaker when available commit falls under the reserve. The HTTP 429 names the commit limit and how to enlarge the pagefile (System Properties → Advanced → Performance → Settings → Advanced → Virtual memory). Raisingindices.breaker.total.limitorXERJ_MAX_PROCESS_MEMORY_MBcannot grant commit Windows will not give. Autoindex already treats that 429 as backpressure and halves bulk concurrency.panic = "abort"is unchanged. Linux and macOS take theNonepath and keep the previous cap, watermark, and 429 sentence. The other abort shapes on #1122 (the memory ceiling tracked with #1032, the WSL2 VM death) are not addressed.Follow-up: use-case harness on c763939
The job
Use-case harnesses (brain + MCP + autoindex)failed on headc763939c. CI tested merge8d4035aof that head intoc16fad587. Phase 5 (xerj brain --freshafter wiping the data dir) printed:The journal/server refusal passed. The
--freshrecovery then exited 1:That failure is not caused by the Windows commit-budget change. The runner is Ubuntu, and Linux still takes the previous cap. It is the torn-snapshot race #2244 hit and #2256 fixed on main: after SIGTERM,
/health/readydrops while the final flush is still writing,rm -rfleavessnapshot.jsonnaming a segment whose.segis already gone, and the catalog search returns HTTP 500. The repair isdrop_local_segments_missing_files(reopen_drops_a_snapshot_segment_whose_file_is_gone). That commit (61c9c70) is not an ancestor ofc16fad587. #2256's own use-case harness passed. No.githubfile was edited.Merged
origin/mainthrough8920c06(no conflicts). The Windows budget behaviour is unchanged. CHANGELOG keeps both Unreleased notes.Evidence
windows_commit_budgeta_pagefileless_16gib_box_is_capped_by_available_commitwindows_commit_budgetample_commit_does_not_move_the_physical_tier_inputwindows_commit_budgeta_failed_commit_probe_does_not_invent_a_tight_capcommit_limit = 0memory_breaker_reasonthe_rss_breaker_message_is_unchanged_without_a_commit_ceilingrefresh_memory_observingcommit_starvation_rejects_writes_and_releases_when_commit_returnsVirtualAlloccurrent_process_memory_bytesprocess_memory_matches_rss_off_windows.segis dropped on open, so a later search does not 500drop_local_segments_missing_filesreopen_drops_a_snapshot_segment_whose_file_is_goneChecks
cargo fmt --all(CI runsrustfmt --checkandclippy -D warnings)cargo build --release -j 32 -p <crate>cargo test -p xerj-common --libandcargo test -p xerj-engine --lib governor::engine/tests/es-compat-yaml/yaml/CHANGELOG.mdUnreleased,limits.max_process_memory_mbrustdoc)docs/CONTRIBUTION_REVIEW.md(independent ofmain; Linux/macOS admission path unchanged and locked by the exact 429 sentence plusprocess_memory_matches_rss_off_windows; Windows refuses writes before they are applied, same 429 the client already retries and shrinks on; no durable-state transition; the prefixreal memory circuit breaker tripped/writes rejected to prevent an out-of-memory killis preserved)Not run:
cargo build --release(fat LTO). ES-YAML was not run: the diff does not change query or wire behaviour except the Windows memory-breaker reason string, which still carries the substrings autoindex matches. No Windows host was available, so the liveGlobalMemoryStatusEx/PagefileUsagepath was not executed; the budget math is unit-tested with the field-report numbers injected. The use-case harness itself was not re-run locally (it needs the CI binary); the unit test that locks the torn-snapshot repair was run. No workflow file was edited.Provenance
--locked, on this Linux VM, after mergingorigin/mainat8920c06. The harness failure text is from GitHub Actions job120583079030(run39997625753).ullTotalPageFileis the commit limit andullAvailPageFileis how much more this process may commit, per theMEMORYSTATUSEXcontract the existing probe already calls. A Windows run where those fields mean something else, or where an allocation larger than the reserve (at most 2 GiB) still aborts the node between samples, would falsify the "backs off instead of aborting" claim. A pagefile that covers RAM does not change the tier; that is tested with injected numbers, not a live pagefile. The harness is assumed to pass once the open-time repair from fix(autoindex,engine): refuse or stream one oversized JSON document (#1122) #2256 is in the merge, because that pull request's use-case job passed and this failure matches the case that repair names. A later run where the dying process unlinks a segment after the new process has already opened the index would still 500, and that is not what this merge changes.