Repository navigation
perf(engine,api): skip empty shards before the finalize gate + O(1) selector lookups (carries #1399 by @todie, #1226) - #1456
Merged
Merged
Conversation
…ndices) selector scans (#1226) Motivation: On a multi-index node (e.g. 7,280 indices) under concurrent bulk write load, `POST /{idx}/_refresh` and `POST /{idx}/_delete_by_query?refresh=true` suffer two compounding bottlenecks: 1. `Index::refresh` and `Index::flush` spawn `do_flush_shard` across all `num_shards` (16 by default), and `do_flush_shard` acquires a permit from the process-wide `flush_finalize_gate()` semaphore BEFORE checking whether the shard has any unflushed documents (`memtable.shard_load(shard_idx).0 == 0`). Every empty shard therefore queues behind in-flight ~600ms segment finalizes from concurrent bulk writers on other indices. 2. `resolve_index_selector` and `resolve_indices_for_op` unconditionally enumerate all open indices (`index_name_list()`, or `list_indices().await` which calls `idx.stats().await` on every open index) even when the selector is a single concrete index name or alias, and 8 read/admin handlers called `list_indices().await` when only index names were needed. Root cause & changes: - `engine/crates/xerj-engine/src/index.rs`: - Skip shards with `memtable.shard_load(shard_idx).0 == 0` in `Index::refresh` and `Index::flush` before spawning tasks, and fast-path empty shards at the top of `do_flush_shard` before `flush_finalize_gate().acquire().await`. - `engine/crates/xerj-engine/src/engine.rs`: - Add `Engine::has_open_index(&self, name: &str) -> bool` for O(1) visible open-index membership lookup. - `engine/crates/xerj-api/src/es_compat.rs`: - Lazily materialize `index_name_list()` in `resolve_index_selector` and `resolve_indices_for_op` only when a wildcard or `_all` pattern is present, using `has_open_index(part)` for concrete names and switching `resolve_indices_for_op` from `list_indices().await` to `index_name_list()`. - Replace 8 name-only `list_indices().await` call sites (`mget`, `search_impl`, `count_docs`, `field_caps`, `msearch_ndjson`, `msearch_template`, `global_field_caps`, and `rollup`) with `index_name_list()`. Verification: - `cargo test -j 2 --manifest-path engine/Cargo.toml -p xerj-engine --lib refresh_and_flush_skip_empty_shards_without_acquiring_finalize_gate` (1 passed) - `cargo test -j 2 --manifest-path engine/Cargo.toml -p xerj-engine --lib has_open_index_tracks_created_and_deleted_indices` (1 passed) - `cargo check -j 2 --manifest-path engine/Cargo.toml -p xerj-api --lib --tests` Fixes #1226
…heck in do_flush_shard #1319 (merged 55232ff) already skips empty shards when Index::refresh / Index::flush build their fan-out, which was the first half of todie's #1399. Rebasing onto main dropped that duplicate; what remains from #1399 on the engine side is the emptiness re-check at the top of do_flush_shard, BEFORE the process-global flush_finalize_gate. It still matters after #1319: - a task spawned for a non-empty shard can find it drained by a concurrent flush by the time it holds its flush_sema permit; - the ingest-triggered flush paths call do_flush_shard directly and never pass through the refresh/flush fan-out filter. The fast path fires on_drained (callers release flush_sema from it) and returns, the same outcome as the existing empty-drain branch minus the wait. Test: #1399's refresh_and_flush_skip_empty_shards_without_acquiring_finalize_gate went through Index::refresh/flush, which #1319's fan-out filter already covers, so it no longer exercised the new code. Replaced by do_flush_shard_skips_finalize_gate_for_empty_shard: exhaust the global gate, call do_flush_shard on an empty shard directly, assert it returns within 2s and on_drained fired. Fails (timeout) without the pre-gate check. cargo test -p xerj-engine --lib do_flush_shard_skips_finalize_gate has_open_index empty_flush_does_not_enqueue -> 3 passed, 0 failed.
|
Thanks for this contribution — we'd like to merge it. Before we can, we need a signed Contributor License Agreement from: @todie It is a one-time thing per contributor, not per pull request. Read CLA.md, then open a small pull request adding your GitHub username to Once it is merged, comment |
Owner
Author
|
@cla-bot check |
|
Re-checking the CLA status for every commit author on this pull request… |
Owner
Author
B06F
|
Independent review: approve, no changes. Thank you, @todie. What I checked:
Merging once CI is green. |
This was referenced Oct 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Carries @todie's #1399 onto current main, reconciled with #1319. Advances #1226. Thank you, @todie: your commit is kept as-is with you as its author, and a follow-up commit from us sits on top of it.
What the code does
Engine (
xerj-engine/src/index.rs,engine.rs)do_flush_shardchecksmemtable.shard_load(shard_idx).0 == 0before it acquires the process-wideflush_finalize_gate. An empty shard fireson_drained(callers release theirflush_semapermit from it) and returns, so it no longer waits behind every other index's in-flight ~600 ms segment finalize.Engine::has_open_index(name) -> bool: an O(1) check for whether a visible index is open.API (
xerj-api/src/es_compat.rs)resolve_index_selectorandresolve_indices_for_opresolve concrete names and aliases withhas_open_index. They build the fullindex_name_list()only when the selector has a wildcard or_all.resolve_indices_for_opno longer callslist_indices().await, which ranstats()on every open index.mget,search_impl,count_docs,field_caps,msearch_ndjson,msearch_template,global_field_caps,rollup) now useindex_name_list()instead oflist_indices().await.How it was reconciled with #1319
#1319 (merged as 55232ff) already skips empty shards when
Index::refreshandIndex::flushbuild their per-shard fan-out. That was the first half of #1399's engine change, so the duplicate went away in the rebase. The rest of #1399's engine change, the check before the gate insidedo_flush_shard, is still needed after #1319:flush_semapermit;do_flush_sharddirectly and never go through the fan-out filter.#1399's test (
refresh_and_flush_skip_empty_shards_without_acquiring_finalize_gate) went throughIndex::refresh/flush, which #1319's filter already covers. On top of #1319 it would pass even without the new check, so it was replaced withdo_flush_shard_skips_finalize_gate_for_empty_shard. The new test holds every permit of the global gate, callsdo_flush_sharddirectly on an empty shard, and asserts that it returns within 2 s and thaton_drainedfired. Without the check it times out.#1226 stays open. Neither PR fixes the contention between foreground and background flushes on the finalize gate (see #1319, "What remains open").
Verification
cargo test -p xerj-engine --lib -- do_flush_shard_skips_finalize_gate has_open_index empty_flush_does_not_enqueue→ 3 passed, 0 failedcargo fmt --checkis clean. CI runs the full suite, including ES-YAML conformance.Supersedes #1399.
🤖 Generated with Claude Code