8000 feat(ai): prototype bounded ONNX embeddings by buger · Pull Request #2 · xerj-org/xerj · GitHub
[go: up one dir, main page]

Skip to content

feat(ai): prototype bounded ONNX embeddings - #2

Merged
xerj-team merged 1 commit into
xerj-org:mainfrom
probelabs:feature/onnx-experimental
Jul 23, 2026
Merged

xerj-team merged 1 commit into
xerj-org:mainfrom
probelabs:feature/onnx-experimental

Conversation

@buger
@buger buger commented Jul 23, 2026
Copy link
Copy Markdown
Contributor

Neural indices previously loaded one Candle model per index, multiplying startup work and resident memory. The experimental ONNX investigation also lived outside a reviewable product change, so its safe execution policy and measured boundaries were not available in-tree.

Share Candle models process-wide by complete NeuralConfig while retaining lazy loading and releasing registry entries after the last user drops. Add an opt-in xerj-ai ONNX FP32 backend using the safe ort API, one shared session, length-aware microbatches, padded-token and queue bounds, input-order restoration, focused planner tests, and a live benchmark example. The default feature set and --embed-mode neural remain Candle; no server or CLI ONNX mode is introduced.

The controlled embedding-layer comparison measured 116.671 docs/s for scheduled ONNX versus 9.045 docs/s for identically planned Candle (12.90x), with minimum same-document cosine 0.9999991655. This is not an end-to-end indexing claim. Measured stripped server deltas were +18.75 MiB for dual backends and +16.43 MiB for ONNX-only. Bundled ort artifacts do not support the project's musl release targets, so the feature remains explicitly experimental.

Validation: default xerj-ai release build and 25 tests; neural release build and 3 model-sharing tests; ONNX release build and 3 scheduler tests; combined neural+ONNX suite (31 passed, 1 ignored); live 256-document example (163.96 scheduled docs/s, 1.87x over singleton, minimum cosine 0.99999905); package formatting and diff checks.

Neural indices previously loaded one Candle model per index, multiplying startup work and resident memory. The experimental ONNX investigation also lived outside a reviewable product change, so its safe execution policy and measured boundaries were not available in-tree.

Share Candle models process-wide by complete NeuralConfig while retaining lazy loading and releasing registry entries after the last user drops. Add an opt-in xerj-ai ONNX FP32 backend using the safe ort API, one shared session, length-aware microbatches, padded-token and queue bounds, input-order restoration, focused planner tests, and a live benchmark example. The default feature set and --embed-mode neural remain Candle; no server or CLI ONNX mode is introduced.

The controlled embedding-layer comparison measured 116.671 docs/s for scheduled ONNX versus 9.045 docs/s for identically planned Candle (12.90x), with minimum same-document cosine 0.9999991655. This is not an end-to-end indexing claim. Measured stripped server deltas were +18.75 MiB for dual backends and +16.43 MiB for ONNX-only. Bundled ort artifacts do not support the project's musl release targets, so the feature remains explicitly experimental.

Validation: default xerj-ai release build and 25 tests; neural release build and 3 model-sharing tests; ONNX release build and 3 scheduler tests; combined neural+ONNX suite (31 passed, 1 ignored); live 256-document example (163.96 scheduled docs/s, 1.87x over singleton, minimum cosine 0.99999905); package formatting and diff checks.
@xerj-team
xerj-team merged commit b2f280e into xerj-org:main Jul 23, 2026
lamardealmaker pushed a commit to lamardealmaker/xerj that referenced this pull request Oct 15, 2026
Advances xerj-org#1226 (partial). Server-side refresh/delete_by_query latency.

## Root cause

`_refresh` and `delete_by_query?refresh=true` both funnel into
`Index::flush()` (the HTTP `_refresh` handler and `run_delete_by_query`'s
paging-precondition flush). `flush()` fans one `do_flush_shard` task out for
EVERY shard (`for shard_idx in 0..n_shards`), and each task first takes a
per-index `flush_sema` permit and then the *process-global*
`flush_finalize_gate` (a handful of permits shared by every index) BEFORE it
discovers the shard has nothing to drain.

Under a multi-index bulk storm that global gate is saturated by other
indices' ingest flushes, so every no-op shard task still queues behind the
whole backlog. A user-visible refresh therefore paid one full finalize-
admission wait PER SHARD even when the memtable held nothing new — which is
exactly the issue's "latency tracks concurrent write load, not index size"
signature (14-16s under the corpus-build storm; 3.4s even on a quiet node).

`Index::refresh()` had the same per-shard fan-out (minus the early
all-empty return it already has).

## What changed

Before spawning a shard's finalize, skip it when the shard's memtable is
empty (`memtable.shard_load(shard_idx).0 == 0`), in both `flush()` and
`refresh()`. An empty shard's finalize is a pure no-op; skipping it keeps it
out of the shared finalize gate entirely. A flush/refresh now admits only the
shards it actually has to drain, and an all-empty memtable does zero finalize
work (just the durability tail — HNSW save, WAL checkpoint).

## Before / after

New regression test `empty_flush_does_not_enqueue_shard_finalize_work`
exhausts an index's own `flush_sema`, then flushes an empty memtable:
- before: the coordinator awaits 16 shard tasks wedged on the exhausted
  semaphore and times out (5s).
- after: no shard task is spawned; the flush returns immediately.

Verified failing-first (times out) with the guard disabled, passing with it.
`cargo test -p xerj-engine --lib flush` (51) green; clippy -D warnings and
fmt clean.

## What remains open (partial)

This bounds a refresh's finalize-admission cost to the shards it touches and
removes no-op refreshes from the shared gate. It does NOT fix the core
contention when the refreshed index's own shards are all non-empty under a
concurrent multi-index storm: the foreground refresh still competes FIFO with
background ingest flushes for the single process-global `flush_finalize_gate`,
with no priority. Fully closing xerj-org#1226 needs the server-side profile the issue
asks for (unknown xerj-org#1) plus a design decision on foreground-vs-background
finalize priority / a reserved foreground lane (unknowns xerj-org#2, xerj-org#3) — a
perf/product tradeoff that stays open.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

0