8000 perf(engine,api): skip empty shards before the finalize gate + O(1) selector lookups (carries #1399 by @todie, #1226) by xerj-org · Pull Request #1456 · xerj-org/xerj · GitHub
[go: up one dir, main page]

Skip to content

perf(engine,api): skip empty shards before the finalize gate + O(1) selector lookups (carries #1399 by @todie, #1226) - #1456

Merged
xerj-org merged 2 commits into
mainfrom
carry/todie-1399
Oct 16, 2026
Merged

xerj-org merged 2 commits into
mainfrom
carry/todie-1399

Conversation

@xerj-org
Copy link
Copy Markdown
Owner

Carries @todie's #1399 onto current main, reconciled with #1319. Advances #1226. Thank you, @todie: your commit is kept as-is with you as its author, and a follow-up commit from us sits on top of it.

What the code does

Engine (xerj-engine/src/index.rs, engine.rs)

  • do_flush_shard checks memtable.shard_load(shard_idx).0 == 0 before it acquires the process-wide flush_finalize_gate. An empty shard fires on_drained (callers release their flush_sema permit from it) and returns, so it no longer waits behind every other index's in-flight ~600 ms segment finalize.
  • New Engine::has_open_index(name) -> bool: an O(1) check for whether a visible index is open.

API (xerj-api/src/es_compat.rs)

  • resolve_index_selector and resolve_indices_for_op resolve concrete names and aliases with has_open_index. They build the full index_name_list() only when the selector has a wildcard or _all. resolve_indices_for_op no longer calls list_indices().await, which ran stats() on every open index.
  • Eight handlers that needed only index names (mget, search_impl, count_docs, field_caps, msearch_ndjson, msearch_template, global_field_caps, rollup) now use index_name_list() instead of list_indices().await.

How it was reconciled with #1319

#1319 (merged as 55232ff) already skips empty shards when Index::refresh and Index::flush build their per-shard fan-out. That was the first half of #1399's engine change, so the duplicate went away in the rebase. The rest of #1399's engine change, the check before the gate inside do_flush_shard, is still needed after #1319:

  • a task spawned for a shard that was non-empty can find it already drained by a concurrent flush by the time it holds its flush_sema permit;
  • the flush paths triggered by ingest call do_flush_shard directly and never go through the fan-out filter.

#1399's test (refresh_and_flush_skip_empty_shards_without_acquiring_finalize_gate) went through Index::refresh/flush, which #1319's filter already covers. On top of #1319 it would pass even without the new check, so it was replaced with do_flush_shard_skips_finalize_gate_for_empty_shard. The new test holds every permit of the global gate, calls do_flush_shard directly on an empty shard, and asserts that it returns within 2 s and that on_drained fired. Without the check it times out.

#1226 stays open. Neither PR fixes the contention between foreground and background flushes on the finalize gate (see #1319, "What remains open").

Verification

  • cargo test -p xerj-engine --lib -- do_flush_shard_skips_finalize_gate has_open_index empty_flush_does_not_enqueue → 3 passed, 0 failed
  • cargo fmt --check is clean. CI runs the full suite, including ES-YAML conformance.

Supersedes #1399.

🤖 Generated with Claude Code

todie and others added 2 commits October 15, 2026 10:24
…ndices) selector scans (#1226)

Motivation:
On a multi-index node (e.g. 7,280 indices) under concurrent bulk write load,
`POST /{idx}/_refresh` and `POST /{idx}/_delete_by_query?refresh=true` suffer
two compounding bottlenecks:
1. `Index::refresh` and `Index::flush` spawn `do_flush_shard` across all
   `num_shards` (16 by default), and `do_flush_shard` acquires a permit from the
   process-wide `flush_finalize_gate()` semaphore BEFORE checking whether the
   shard has any unflushed documents (`memtable.shard_load(shard_idx).0 == 0`).
   Every empty shard therefore queues behind in-flight ~600ms segment finalizes
   from concurrent bulk writers on other indices.
2. `resolve_index_selector` and `resolve_indices_for_op` unconditionally
   enumerate all open indices (`index_name_list()`, or `list_indices().await`
   which calls `idx.stats().await` on every open index) even when the selector
   is a single concrete index name or alias, and 8 read/admin handlers called
   `list_indices().await` when only index names were needed.

Root cause & changes:
- `engine/crates/xerj-engine/src/index.rs`:
  - Skip shards with `memtable.shard_load(shard_idx).0 == 0` in `Index::refresh`
    and `Index::flush` before spawning tasks, and fast-path empty shards at the
    top of `do_flush_shard` before `flush_finalize_gate().acquire().await`.
- `engine/crates/xerj-engine/src/engine.rs`:
  - Add `Engine::has_open_index(&self, name: &str) -> bool` for O(1) visible
    open-index membership lookup.
- `engine/crates/xerj-api/src/es_compat.rs`:
  - Lazily materialize `index_name_list()` in `resolve_index_selector` and
    `resolve_indices_for_op` only when a wildcard or `_all` pattern is present,
    using `has_open_index(part)` for concrete names and switching `resolve_indices_for_op`
    from `list_indices().await` to `index_name_list()`.
  - Replace 8 name-only `list_indices().await` call sites (`mget`, `search_impl`,
    `count_docs`, `field_caps`, `msearch_ndjson`, `msearch_template`,
    `global_field_caps`, and `rollup`) with `index_name_list()`.

Verification:
- `cargo test -j 2 --manifest-path engine/Cargo.toml -p xerj-engine --lib refresh_and_flush_skip_empty_shards_without_acquiring_finalize_gate` (1 passed)
- `cargo test -j 2 --manifest-path engine/Cargo.toml -p xerj-engine --lib has_open_index_tracks_created_and_deleted_indices` (1 passed)
- `cargo check -j 2 --manifest-path engine/Cargo.toml -p xerj-api --lib --tests`

Fixes #1226
…heck in do_flush_shard

#1319 (merged 55232ff) already skips empty shards when Index::refresh /
Index::flush build their fan-out, which was the first half of todie's #1399.
Rebasing onto main dropped that duplicate; what remains from #1399 on the
engine side is the emptiness re-check at the top of do_flush_shard, BEFORE
the process-global flush_finalize_gate. It still matters after #1319:

- a task spawned for a non-empty shard can find it drained by a concurrent
  flush by the time it holds its flush_sema permit;
- the ingest-triggered flush paths call do_flush_shard directly and never
  pass through the refresh/flush fan-out filter.

The fast path fires on_drained (callers release flush_sema from it) and
returns, the same outcome as the existing empty-drain branch minus the wait.

Test: #1399's refresh_and_flush_skip_empty_shards_without_acquiring_finalize_gate
went through Index::refresh/flush, which #1319's fan-out filter already
covers, so it no longer exercised the new code. Replaced by
do_flush_shard_skips_finalize_gate_for_empty_shard: exhaust the global gate,
call do_flush_shard on an empty shard directly, assert it returns within 2s
and on_drained fired. Fails (timeout) without the pre-gate check.

cargo test -p xerj-engine --lib do_flush_shard_skips_finalize_gate has_open_index
empty_flush_does_not_enqueue -> 3 passed, 0 failed.
@cla-bot
cla-bot Bot commented Oct 15, 2026
Copy link
Copy Markdown

Thanks for this contribution — we'd like to merge it.

Before we can, we need a signed Contributor License Agreement from: @todie

It is a one-time thing per contributor, not per pull request. Read CLA.md, then open a small pull request adding your GitHub username to .contributors — that pull request is your signature, since it is made from your own account.

Once it is merged, comment @cla-bot check here and this check will go green. If you believe you are already covered, comment @cla-bot check and it will re-verify.

@xerj-org
Copy link
Copy Markdown
Owner Author

@cla-bot check

@cla-bot cla-bot Bot added the cla-signed label Oct 15, 2026
@cla-bot
cla-bot Bot commented Oct 15, 2026
Copy link
Copy Markdown

Re-checking the CLA status for every commit author on this pull request…

@xerj-org
Copy link
Copy Markdown
Owner Author
B06F

Independent review: approve, no changes. Thank you, @todie.

What I checked:

Merging once CI is green.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0