Repository navigation
fix(engine): apply persisted default_pipeline after restart - #2296
Merged
Merged
Conversation
An index created with index.default_pipeline kept the binding in settings.json, so GET /_settings still reported it after a clean restart and the pipeline definition survived in cluster state. Ordinary writes stopped running the pipeline. Engine::default_pipeline_for reads only the in-memory index_settings map, and that map was filled on create and on PUT /_settings. Boot, failed-index retry, and snapshot restore opened the index and never copied settings.json back. Opening an existing index now reloads that file into the map before the index is visible. A missing file clears a stale entry so a restored snapshot without settings does not keep the previous binding. A template pipeline copied onto the index at create (#2281) comes back with the same file. A template registered after the index exists still does not attach on open. index.final_pipeline is not a setting this build reads. The regression test failed before the change: after Engine::new on the same data dir, default_pipeline_for returned None while GET /_settings still showed restart-copy. It passes after. An AI coding agent wrote this change. Fixes #2294. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
Keep both Unreleased Fixed notes: the persisted default_pipeline restart fix (#2294) and the _refresh stall fix (#2234). engine.rs auto-merged. rehydrate_index_settings_from_disk still runs on boot, failed-index retry, and snapshot restore, on top of the refresh, Windows commit-cap, and flood-stage changes from main. cargo fmt --all made no further edits. Verified with rustc 1.97.1 from engine/: - cargo test --profile ci-test -p xerj-api --test default_pipeline_survives_restart: 1 passed - cargo test --profile ci-test -p xerj-engine -p xerj-api -- --test-threads=2: 2309 passed, 0 failed, 26 ignored Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
Main moved again while CI ran (#2298, #2300, #2302). Keep both Unreleased Fixed notes: the persisted default_pipeline restart fix (#2294) and the large-JSON ingest bound (#1122). No engine.rs conflict. rehydrate_index_settings_from_disk is unchanged. cargo fmt --all made no further edits. Verified with rustc 1.97.1 from engine/: - cargo test --profile ci-test -p xerj-api --test default_pipeline_survives_restart: 1 passed - cargo test --profile ci-test -p xerj-engine -p xerj-api -- --test-threads=2: 2311 passed, 0 failed, 26 ignored Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
Owner
Author
cursor Bot
pushed a commit
that referenced
this pull request
Oct 24, 2026
Nine pull requests landed on main after the rc.99 cut. They are cited in [1.0.0-rc.99] - 2026-10-23, and [Unreleased] stays empty. - #2245 fixes #2234: _refresh no longer waits on a background merge. - #2250 fixes #1612: autoindex estimates neural embedding time, and docs say --embed-mode is fixed at ingest. - #2296 closes #2294: a persisted index.default_pipeline runs again after a clean restart. - #2298: DeepSeek Harness profile bundle in dsh/, README links, and landing/brandbook/xerj-mark-400.png. dsh/package.json is 1.0.0-rc.99. - #2300 and #2301: Cline is marked tested in the EN/KO/ZH README tables, with a field report. The run was against v1.0.0-rc.98. - #2302 improves #1122, which stays open: a bulk reserves memory before parsing, and uncompressed JSON over 64 MiB is streamed. - #2303 is part of #1170, which stays open: a document write is acknowledged once a quorum of copies confirms. - #2308 fixes #2305: a semantic query over a wildcard, alias, or comma list skips indices that have no semantic_text field. The window since v1.0.0-rc.98 is 91 merged pull requests. The 18 crate versions in engine/Cargo.lock stay 1.0.0-rc.99. Landing footers that this merge touched stay V1.0.0-RC.99. /product now says rc.99 lists xerj hub next to xerj ask when the build has a hub signing key (#2282); the rc.98 capture of that page is otherwise unchanged. #2312 merged to corpus-hub, not main, so it is not in this cut. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
Merged
cursor Bot
pushed a commit
that referenced
this pull request
Oct 24, 2026
Motivation: #2296 (persisted default_pipeline after restart) and #2250 (neural ingest estimate) landed on main after the previous merge, and the branch conflicted again. Resolution: keep every Unreleased changelog entry, including the proxy semantic resume (#2306), the default_pipeline restart fix, and the neural ingest estimate. EmbeddingConfig keeps both runtime cells: the confirmed proxy width and the neural ingest probe. No .github files were edited. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
cursor Bot
pushed a commit
that referenced
this pull request
Oct 24, 2026
The only conflict was CHANGELOG.md. engine.rs and landing/llms-full.txt auto-merged. Both behaviours stay. #1981 (this branch): a flat RSS at or above the watermark releases the breaker after 2 seconds and does not re-arm inside the 4 MiB band. While that latch stays engaged, drain requests double from 2 seconds up to 15 minutes unless RSS falls by more than 4 MiB. corpus index reports eta_quality=stalled at 0/N and after 10 minutes of silence on a phase with no denominator. main, kept alongside that: - #2296 / #2294: an existing index's persisted index.default_pipeline is copied from settings.json back into the in-memory map on boot, failed-index retry, and snapshot restore. - #2303: a single-document write is acknowledged at a quorum of copies, with the replica-sync lock on the engine. - #2250 / #1612: neural autoindex prints an estimated embedding time, and --embed-mode stays fixed at ingest. llms-full.txt keeps that text and the 0/N stall sentence. - #2308 / #2305 and #2311 came in with no overlap on the drain schedule. CHANGELOG keeps the #1981 entry and main's #2294 default_pipeline entry. No .github file was edited. Verified (rustc 1.97.1 (8bab26f4f 2026-07-14), from engine/): - cargo +1.97.1 fmt -p xerj-engine -p xerj-autoindex -- --check: exit 0 - RUST_MIN_STACK=32MiB cargo +1.97.1 test --no-fail-fast -j 2 -p xerj-engine -p xerj-autoindex -- --test-threads=2: exit 0, 3118 passed, 0 failed, 29 ignored. Includes flat_rss_above_the_watermark_releases_and_does_not_rearm, drain_requests_back_off_when_rss_does_not_fall, rss_noise_around_a_flat_footprint_still_releases, commit_starvation_holds_a_flat_footprint_until_commit_returns, and a_phase_at_zero_reports_a_stall_once_nothing_completes. Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes, and why
Fixes #2294.
An index created with
index.default_pipelinekept that binding insettings.json. After a clean restart,GET /_settingsstill reported it (the response falls back to the file when the in-memory map is empty) and the pipeline definition was still in cluster state, but an ordinary document write no longer ran the pipeline. Explicit?pipeline=still worked.Engine::default_pipeline_foris the one lookup_docand_bulkshare, and it reads onlyengine.index_settings. Create andPUT /_settingsfill that map. Boot, failed-index retry, and snapshot restore calledIndex::open, which loadssettings.jsoninto the index, and never copied it into the map. The map started empty, so the lookup returnedNonefor an index that already existed and did not fall through to a template.Opening an existing index now copies
settings.jsoninto the map before the index is visible. A missing file, or JSONnull, drops a stale entry so a restored snapshot that was created without settings does not keep the previous process's binding. A template pipeline stored on the index at create (#2281) comes back with the same file. A template registered after the index exists still does not attach on open.index.final_pipelineis not a setting this build reads, so nothing was added for it.A
PUT /{index}/_settingsthat setsdefault_pipelineafter create still updates only the in-memory display map. That write was never stored insettings.json, so a restart still drops it andGET /_settingsstops reporting it. That is a separate persistence gap from #2294, whose reproduction sets the pipeline in the create body and still sees it inGET /_settingsafter restart.Evidence
Before the fix,
cargo +1.97.1 test -p xerj-api --test default_pipeline_survives_restartfailed:GET /_settingshad already returned the binding. After the change the same test passes (0.07s), including dotted and nested spellings, a_bulkwrite, a template pipeline copied at create, and an index that must stay unbound when a matching template is registered later.Also run, rustc 1.97.1:
cargo +1.97.1 test -p xerj-api --test index_template_applies_semantic_text— 6 passedcargo +1.97.1 test -p xerj-api --test fail_closed_config_http bulk_honours_index_default_pipeline— passedcargo +1.97.1 clippy -p xerj-engine --lib -- -D warnings— passedcargo +1.97.1 clippy -p xerj-api --test default_pipeline_survives_restart -- -D warnings— passedcargo +1.97.1 fmt --all -- --check— passedcargo +1.97.1 build --profile quick -j 4 -p xerj-api -p xerj-engine— finished, exit 0index.default_pipelineruns after restartsettings.jsoncopied intoindex_settingson opendefault_pipeline_survives_restartdefault_pipeline_forwasNonePUT /_settingsafter create is still not persistedfrom-template-probedefault_pipeline_forwhen the index existsplain-probecopied_markerabsentChecks
cargo fmt --all(CI runsrustfmt --checkandclippy -D warnings)cargo build --profile quick -j 4 -p xerj-api -p xerj-engine(the project's verification profile; fat-LTO--releasewas not run)cargo test -p xerj-api --test default_pipeline_survives_restartpasses, plus the template and bulk default-pipeline tests named above. The fullcargo test -p xerj-api/ workspace suite was not run locally.CHANGELOG.mdunder Unreleased / Fixed. Nollms.txtchange; the capability was already the contract and was wrong only after restart.settings.json(unchanged shape, read on open, not rewritten). No new field, no migration, no recipe, no performance claim, no feature gate. Failure to read the file leaves the in-memory map unchanged and does not fail the open;Index::openalready refuses an unparseablesettings.jsonbefore this runs.Not run: workspace
cargo test(CI Build + Test), the ES-YAML gate,cargo clippy --workspace --all-targets, and a fat-LTO--releasebuild.Provenance
rehydrate_index_settings_from_diskexisted, and the same test passed after it.index.final_pipelineremains unsupported.