Repository navigation
feat(ai): prototype bounded ONNX embeddings - #2
Merged
Merged
Conversation
Neural indices previously loaded one Candle model per index, multiplying startup work and resident memory. The experimental ONNX investigation also lived outside a reviewable product change, so its safe execution policy and measured boundaries were not available in-tree. Share Candle models process-wide by complete NeuralConfig while retaining lazy loading and releasing registry entries after the last user drops. Add an opt-in xerj-ai ONNX FP32 backend using the safe ort API, one shared session, length-aware microbatches, padded-token and queue bounds, input-order restoration, focused planner tests, and a live benchmark example. The default feature set and --embed-mode neural remain Candle; no server or CLI ONNX mode is introduced. The controlled embedding-layer comparison measured 116.671 docs/s for scheduled ONNX versus 9.045 docs/s for identically planned Candle (12.90x), with minimum same-document cosine 0.9999991655. This is not an end-to-end indexing claim. Measured stripped server deltas were +18.75 MiB for dual backends and +16.43 MiB for ONNX-only. Bundled ort artifacts do not support the project's musl release targets, so the feature remains explicitly experimental. Validation: default xerj-ai release build and 25 tests; neural release build and 3 model-sharing tests; ONNX release build and 3 scheduler tests; combined neural+ONNX suite (31 passed, 1 ignored); live 256-document example (163.96 scheduled docs/s, 1.87x over singleton, minimum cosine 0.99999905); package formatting and diff checks.
This was referenced Jul 25, 2026
6 of 8 tasks
This was referenced Aug 4, 2026
This was referenced Aug 17, 2026
This was referenced Aug 20, 2026
Merged
2 tasks done
lamardealmaker
pushed a commit
to lamardealmaker/xerj
that referenced
this pull request
Oct 15, 2026
Advances xerj-org#1226 (partial). Server-side refresh/delete_by_query latency. ## Root cause `_refresh` and `delete_by_query?refresh=true` both funnel into `Index::flush()` (the HTTP `_refresh` handler and `run_delete_by_query`'s paging-precondition flush). `flush()` fans one `do_flush_shard` task out for EVERY shard (`for shard_idx in 0..n_shards`), and each task first takes a per-index `flush_sema` permit and then the *process-global* `flush_finalize_gate` (a handful of permits shared by every index) BEFORE it discovers the shard has nothing to drain. Under a multi-index bulk storm that global gate is saturated by other indices' ingest flushes, so every no-op shard task still queues behind the whole backlog. A user-visible refresh therefore paid one full finalize- admission wait PER SHARD even when the memtable held nothing new — which is exactly the issue's "latency tracks concurrent write load, not index size" signature (14-16s under the corpus-build storm; 3.4s even on a quiet node). `Index::refresh()` had the same per-shard fan-out (minus the early all-empty return it already has). ## What changed Before spawning a shard's finalize, skip it when the shard's memtable is empty (`memtable.shard_load(shard_idx).0 == 0`), in both `flush()` and `refresh()`. An empty shard's finalize is a pure no-op; skipping it keeps it out of the shared finalize gate entirely. A flush/refresh now admits only the shards it actually has to drain, and an all-empty memtable does zero finalize work (just the durability tail — HNSW save, WAL checkpoint). ## Before / after New regression test `empty_flush_does_not_enqueue_shard_finalize_work` exhausts an index's own `flush_sema`, then flushes an empty memtable: - before: the coordinator awaits 16 shard tasks wedged on the exhausted semaphore and times out (5s). - after: no shard task is spawned; the flush returns immediately. Verified failing-first (times out) with the guard disabled, passing with it. `cargo test -p xerj-engine --lib flush` (51) green; clippy -D warnings and fmt clean. ## What remains open (partial) This bounds a refresh's finalize-admission cost to the shards it touches and removes no-op refreshes from the shared gate. It does NOT fix the core contention when the refreshed index's own shards are all non-empty under a concurrent multi-index storm: the foreground refresh still competes FIFO with background ingest flushes for the single process-global `flush_finalize_gate`, with no priority. Fully closing xerj-org#1226 needs the server-side profile the issue asks for (unknown xerj-org#1) plus a design decision on foreground-vs-background finalize priority / a reserved foreground lane (unknowns xerj-org#2, xerj-org#3) — a perf/product tradeoff that stays open.
Merged
4 of 8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Neural indices previously loaded one Candle model per index, multiplying startup work and resident memory. The experimental ONNX investigation also lived outside a reviewable product change, so its safe execution policy and measured boundaries were not available in-tree.
Share Candle models process-wide by complete NeuralConfig while retaining lazy loading and releasing registry entries after the last user drops. Add an opt-in xerj-ai ONNX FP32 backend using the safe ort API, one shared session, length-aware microbatches, padded-token and queue bounds, input-order restoration, focused planner tests, and a live benchmark example. The default feature set and --embed-mode neural remain Candle; no server or CLI ONNX mode is introduced.
The controlled embedding-layer comparison measured 116.671 docs/s for scheduled ONNX versus 9.045 docs/s for identically planned Candle (12.90x), with minimum same-document cosine 0.9999991655. This is not an end-to-end indexing claim. Measured stripped server deltas were +18.75 MiB for dual backends and +16.43 MiB for ONNX-only. Bundled ort artifacts do not support the project's musl release targets, so the feature remains explicitly experimental.
Validation: default xerj-ai release build and 25 tests; neural release build and 3 model-sharing tests; ONNX release build and 3 scheduler tests; combined neural+ONNX suite (31 passed, 1 ignored); live 256-document example (163.96 scheduled docs/s, 1.87x over singleton, minimum cosine 0.99999905); package formatting and diff checks.