8000 fix(engine): apply persisted default_pipeline after restart by xerj-org · Pull Request #2296 · xerj-org/xerj · GitHub
[go: up one dir, main page]

Skip to content

fix(engine): apply persisted default_pipeline after restart - #2296

Merged
xerj-org merged 3 commits into
mainfrom
cursor/default-pipeline-restart-f457
Oct 24, 2026
Merged

xerj-org merged 3 commits into
mainfrom
cursor/default-pipeline-restart-f457

Conversation

@xerj-org
Copy link
Copy Markdown
Owner

What this changes, and why

Fixes #2294.

An index created with index.default_pipeline kept that binding in settings.json. After a clean restart, GET /_settings still reported it (the response falls back to the file when the in-memory map is empty) and the pipeline definition was still in cluster state, but an ordinary document write no longer ran the pipeline. Explicit ?pipeline= still worked.

Engine::default_pipeline_for is the one lookup _doc and _bulk share, and it reads only engine.index_settings. Create and PUT /_settings fill that map. Boot, failed-index retry, and snapshot restore called Index::open, which loads settings.json into the index, and never copied it into the map. The map started empty, so the lookup returned None for an index that already existed and did not fall through to a template.

Opening an existing index now copies settings.json into the map before the index is visible. A missing file, or JSON null, drops a stale entry so a restored snapshot that was created without settings does not keep the previous process's binding. A template pipeline stored on the index at create (#2281) comes back with the same file. A template registered after the index exists still does not attach on open.

index.final_pipeline is not a setting this build reads, so nothing was added for it.

A PUT /{index}/_settings that sets default_pipeline after create still updates only the in-memory display map. That write was never stored in settings.json, so a restart still drops it and GET /_settings stops reporting it. That is a separate persistence gap from #2294, whose reproduction sets the pipeline in the create body and still sees it in GET /_settings after restart.

Evidence

Before the fix, cargo +1.97.1 test -p xerj-api --test default_pipeline_survives_restart failed:

assertion `left == right` failed: the in-memory binding must be rebuilt from settings.json
  left: None
  right: Some("restart-copy")

GET /_settings had already returned the binding. After the change the same test passes (0.07s), including dotted and nested spellings, a _bulk write, a template pipeline copied at create, and an index that must stay unbound when a matching template is registered later.

Also run, rustc 1.97.1:

  • cargo +1.97.1 test -p xerj-api --test index_template_applies_semantic_text — 6 passed
  • cargo +1.97.1 test -p xerj-api --test fail_closed_config_http bulk_honours_index_default_pipeline — passed
  • cargo +1.97.1 clippy -p xerj-engine --lib -- -D warnings — passed
  • cargo +1.97.1 clippy -p xerj-api --test default_pipeline_survives_restart -- -D warnings — passed
  • cargo +1.97.1 fmt --all -- --check — passed
  • cargo +1.97.1 build --profile quick -j 4 -p xerj-api -p xerj-engine — finished, exit 0
Claim Source Test Negative control Result Gap
Persisted index.default_pipeline runs after restart settings.json copied into index_settings on open default_pipeline_survives_restart same test before the copy, default_pipeline_for was None pass on final head PUT /_settings after create is still not persisted
Template pipeline stored at create survives restart same file, written by #2281's merge same test, from-template-probe — pass —
A later template does not attach to an existing index early return in default_pipeline_for when the index exists same test, plain-probe — pass, copied_marker absent —

Checks

  • cargo fmt --all (CI runs rustfmt --check and clippy -D warnings)
  • Scoped build of the crates touched: cargo build --profile quick -j 4 -p xerj-api -p xerj-engine (the project's verification profile; fat-LTO --release was not run)
  • cargo test -p xerj-api --test default_pipeline_survives_restart passes, plus the template and bulk default-pipeline tests named above. The full cargo test -p xerj-api / workspace suite was not run locally.
  • ES-YAML conformance suite at 0 failed — not run. This is not a new request shape; it is the existing default-pipeline contract across a process restart, which the YAML runner does not model. The regression is the HTTP test above.
  • New ES-compatible behaviour has a matching YAML case — N/A for the same reason.
  • Docs updated: CHANGELOG.md under Unreleased / Fixed. No llms.txt change; the capability was already the contract and was wrong only after restart.
  • Contribution review: durable state is the existing settings.json (unchanged shape, read on open, not rewritten). No new field, no migration, no recipe, no performance claim, no feature gate. Failure to read the file leaves the in-memory map unchanged and does not fail the open; Index::open already refuses an unparseable settings.json before this runs.
  • Operational-recipe audit: N/A. No operational document or runnable recipe changed.

Not run: workspace cargo test (CI Build + Test), the ES-YAML gate, cargo clippy --workspace --all-targets, and a fat-LTO --release build.

Provenance

  • Written by: AI agent (Cursor Grok 4.7), on the account opening this PR.
  • Verified: the commands and outputs in Evidence. The failing assertion was observed on this branch before rehydrate_index_settings_from_disk existed, and the same test passed after it.
  • Assumed: snapshot restore and failed-index retry take the same helper as boot. Those two call sites are not covered by the restart test. index.final_pipeline remains unsupported.
Open in Web Open in Cursor 

An index created with index.default_pipeline kept the binding in
settings.json, so GET /_settings still reported it after a clean
restart and the pipeline definition survived in cluster state. Ordinary
writes stopped running the pipeline. Engine::default_pipeline_for reads
only the in-memory index_settings map, and that map was filled on
create and on PUT /_settings. Boot, failed-index retry, and snapshot
restore opened the index and never copied settings.json back.

Opening an existing index now reloads that file into the map before the
index is visible. A missing file clears a stale entry so a restored
snapshot without settings does not keep the previous binding. A template
pipeline copied onto the index at create (#2281) comes back with the
same file. A template registered after the index exists still does not
attach on open. index.final_pipeline is not a setting this build reads.

The regression test failed before the change: after Engine::new on the
same data dir, default_pipeline_for returned None while GET /_settings
still showed restart-copy. It passes after.

An AI coding agent wrote this change.

Fixes #2294.

Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
@cla-bot cla-bot Bot added the cla-signed label Oct 23, 2026
cursoragent and others added 2 commits October 24, 2026 00:22
Keep both Unreleased Fixed notes: the persisted default_pipeline restart
fix (#2294) and the _refresh stall fix (#2234). engine.rs auto-merged.
rehydrate_index_settings_from_disk still runs on boot, failed-index
retry, and snapshot restore, on top of the refresh, Windows commit-cap,
and flood-stage changes from main.

cargo fmt --all made no further edits.
Verified with rustc 1.97.1 from engine/:
- cargo test --profile ci-test -p xerj-api --test default_pipeline_survives_restart: 1 passed
- cargo test --profile ci-test -p xerj-engine -p xerj-api -- --test-threads=2: 2309 passed, 0 failed, 26 ignored

Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
Main moved again while CI ran (#2298, #2300, #2302). Keep both
Unreleased Fixed notes: the persisted default_pipeline restart fix
(#2294) and the large-JSON ingest bound (#1122). No engine.rs conflict.
rehydrate_index_settings_from_disk is unchanged.

cargo fmt --all made no further edits.
Verified with rustc 1.97.1 from engine/:
- cargo test --profile ci-test -p xerj-api --test default_pipeline_survives_restart: 1 passed
- cargo test --profile ci-test -p xerj-engine -p xerj-api -- --test-threads=2: 2311 passed, 0 failed, 26 ignored

Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
@xerj-org
Copy link
Copy Markdown
Owner Author

Xerj Merger made the fix: an existing index's persisted index.default_pipeline is now rebuilt from settings.json on boot, failed-index retry and snapshot restore, so ordinary writes run it again after a restart. Reviewed the diff; all checks green on 591878c. Fixes #2294.

@xerj-org
xerj-org merged commit d53bc2e into main Oct 24, 2026
23 checks passed
cursor Bot pushed a commit that referenced this pull request Oct 24, 2026
Nine pull requests landed on main after the rc.99 cut. They are cited
in [1.0.0-rc.99] - 2026-10-23, and [Unreleased] stays empty.

- #2245 fixes #2234: _refresh no longer waits on a background merge.
- #2250 fixes #1612: autoindex estimates neural embedding time, and
  docs say --embed-mode is fixed at ingest.
- #2296 closes #2294: a persisted index.default_pipeline runs again
  after a clean restart.
- #2298: DeepSeek Harness profile bundle in dsh/, README links, and
  landing/brandbook/xerj-mark-400.png. dsh/package.json is 1.0.0-rc.99.
- #2300 and #2301: Cline is marked tested in the EN/KO/ZH README tables,
  with a field report. The run was against v1.0.0-rc.98.
- #2302 improves #1122, which stays open: a bulk reserves memory before
  parsing, and uncompressed JSON over 64 MiB is streamed.
- #2303 is part of #1170, which stays open: a document write is
  acknowledged once a quorum of copies confirms.
- #2308 fixes #2305: a semantic query over a wildcard, alias, or comma
  list skips indices that have no semantic_text field.

The window since v1.0.0-rc.98 is 91 merged pull requests. The 18 crate
versions in engine/Cargo.lock stay 1.0.0-rc.99. Landing footers that
this merge touched stay V1.0.0-RC.99. /product now says rc.99 lists
xerj hub next to xerj ask when the build has a hub signing key (#2282);
the rc.98 capture of that page is otherwise unchanged.

#2312 merged to corpus-hub, not main, so it is not in this cut.

Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
@cursor cursor Bot mentioned this pull request Oct 24, 2026
cursor Bot pushed a commit that referenced this pull request Oct 24, 2026
Motivation: #2296 (persisted default_pipeline after restart) and #2250
(neural ingest estimate) landed on main after the previous merge, and
the branch conflicted again.

Resolution: keep every Unreleased changelog entry, including the proxy
semantic resume (#2306), the default_pipeline restart fix, and the
neural ingest estimate. EmbeddingConfig keeps both runtime cells: the
confirmed proxy width and the neural ingest probe.

No .github files were edited.

Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
cursor Bot pushed a commit that referenced this pull request Oct 24, 2026
The only conflict was CHANGELOG.md. engine.rs and landing/llms-full.txt
auto-merged. Both behaviours stay.

#1981 (this branch): a flat RSS at or above the watermark releases the
breaker after 2 seconds and does not re-arm inside the 4 MiB band.
While that latch stays engaged, drain requests double from 2 seconds
up to 15 minutes unless RSS falls by more than 4 MiB. corpus index
reports eta_quality=stalled at 0/N and after 10 minutes of silence on
a phase with no denominator.

main, kept alongside that:
- #2296 / #2294: an existing index's persisted index.default_pipeline
  is copied from settings.json back into the in-memory map on boot,
  failed-index retry, and snapshot restore.
- #2303: a single-document write is acknowledged at a quorum of copies,
  with the replica-sync lock on the engine.
- #2250 / #1612: neural autoindex prints an estimated embedding time,
  and --embed-mode stays fixed at ingest. llms-full.txt keeps that
  text and the 0/N stall sentence.
- #2308 / #2305 and #2311 came in with no overlap on the drain schedule.

CHANGELOG keeps the #1981 entry and main's #2294 default_pipeline
entry. No .github file was edited.

Verified (rustc 1.97.1 (8bab26f4f 2026-07-14), from engine/):
- cargo +1.97.1 fmt -p xerj-engine -p xerj-autoindex -- --check: exit 0
- RUST_MIN_STACK=32MiB cargo +1.97.1 test --no-fail-fast -j 2
  -p xerj-engine -p xerj-autoindex -- --test-threads=2: exit 0,
  3118 passed, 0 failed, 29 ignored. Includes
  flat_rss_above_the_watermark_releases_and_does_not_rearm,
  drain_requests_back_off_when_rss_does_not_fall,
  rss_noise_around_a_flat_footprint_still_releases,
  commit_starvation_holds_a_flat_footprint_until_commit_returns, and
  a_phase_at_zero_reports_a_stall_once_nothing_completes.

Co-authored-by: xerj-org <xerj-org@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Existing index retains default_pipeline in GET settings but stops applying it after clean restart

2 participants

0