8000 xerj/ROADMAP.md at ux/console-review · hpahm/xerj · GitHub
[go: up one dir, main page]

Skip to content

Latest commit

 

History

History
194 lines (159 loc) · 36.8 KB

ROADMAP.md

File metadata and controls

194 lines (159 loc) · 36.8 KB
8000

XERJ Roadmap

This roadmap tracks capabilities that are planned but not yet fully implemented, so the project's public claims stay honest about what ships today versus what is coming. Status is verified against the actual code and by real API requests to the release binary, not aspirational.

Last reviewed: 2026-09-29 (against v1.0.0-rc.78 and main). Statuses trace to issues, merged PRs, the CHANGELOG, and the conformance suite; items carried forward from the 2026-07-12 review without fresh live verification are marked as such. This review line is machine-checked: docs_capability_lists fails the build if a release is cut without re-reviewing this file (issue #298 — closed as abandoned 2026-09-29; the machine check, not the issue, enforces the cadence now). This pass is the rc.78 release-cut roll: Next release points at rc.79, the riding-to list folded into the rc.78 cut record below, and the open-defects shortlist was re-verified against the live tracker at cut time (#1031 and #1032 open; #1030 rides with them on the milestone) — the lesson of 2026-09-21, when several entries went stale within hours of that review, is why the shortlist is checked live at the cut rather than desk-carried. Later the same day the tracker was emptied: everything open after the cut — #1038, #1030, #1031, #1032 — was closed (the two shipped-half epics are in the rc.78 record below, the two defects are recorded below as deferred, not fixed), leaving zero open issues. The 2026-09-26 desk review (PR #1036) stands as recorded: it closed the CHANGELOG-gap GA item (the rc.19–rc.70 backfill, PR #1035), marked the stage-2 object-storage item done (#965 wired in rc.77), and corrected the mail-ingest memory line to the post-#1002 reality. The Shipping today claims were last live-verified against rc.76 (unchanged by this pass), and The zero-token direction below was verified separately on 2026-09-18, against main @ 4d8dadbf.

Follow the roadmap

  • This file is authoritative. If any other surface disagrees with it, this file wins — and that disagreement is a bug worth an issue.
  • Milestones — the release-by-release view. Every open issue is triaged onto a milestone; the next RC's milestone is the short-term roadmap.
  • Project board — live status of every open item.
  • Pinned plan tracker #1065 — the working index of the ten-item program to 1.0 (discussion #1054). This file remains authoritative; the tracker indexes the filed issues.

Shipping today (for context)

These are implemented and exercised by real API requests / the test suite / benchmarks:

  • Elasticsearch REST wire compatibility (1,366 / 1,369 ES-YAML conformance cases; the gate on every change is 0 failed, and the case count grows as cases are added — read the current number off CI, not off this file).
  • Full-text search (BM25) and 50 query types. Neither the list nor the number is maintained by hand here: the list is generated from xerj_query::parser::SUPPORTED_QUERY_TYPES, printed in full in engine/README.md and llms-full.txt, and pinned to parse_query's dispatch table by parser::tests::dispatch_table_matches_capability_manifest; the number above sits in a machine-checked region pinned to that constant's length by docs_capability_lists::published_capability_counts_match_the_constants. A further 2 keys — has_child and has_parent — are recognised and rejected with a 400, and are listed as such in the same places (issue #211). Honest caveat, unchanged: that count is the dispatch surface — every name on it parses, plans and executes, which is not a claim that every one is semantically faithful to ES. The known divergences are enumerated under Known partials below, and the ES-YAML conformance suite is the measured answer.
  • Aggregations: 62 types, likewise generated from xerj_engine::aggs::SUPPORTED_AGG_TYPES, printed in engine/README.md and llms-full.txt, and pinned by the same count test. This includes the full pipeline family. weighted_avg is not in SUPPORTED_AGG_TYPES — see Known partials. Exactness, precisely. No probabilistic sketch sits in the metric path: cardinality is a true distinct count rather than an HLL estimate, and terms doc_count is precise. Two deliberate exceptions, stated the same way in engine/README.md and llms-full.txt: (1) the sampling family is a sample by definition — run_sampler sorts the matched documents by _score and keeps the first shard_size (default 200), so every sub-aggregation under sampler, random_sampler or diversified_sampler is computed over that slice rather than the whole match set, diversified_sampler additionally caps documents per field value, and random_sampler shares the sampler implementation and ignores ES's probability (an accepted-and-ignored input, #204); (2) percentiles with the hdr option returns HdrHistogram-quantized values, deliberately, so ES's own outputs reproduce — the default tdigest path sorts every value and interpolates instead.
  • Dense-vector kNN (knn query and ES 8.x top-level knn): unfiltered kNN on a full-precision cosine field (≥1,024 docs) is served by a persisted HNSW graph with exact rescoring — measured recall@10 1.00 on the official bench query, 100-probe mean 0.976 (ES 8.13.4 same protocol: 0.937); filtered/nested kNN, non-cosine similarity, SQ8 fields, and small indexes run the exact brute-force scan (cosine mapped to (1+cos)/2).
  • Hybrid search — BM25 + kNN combined in a single request via the hybrid query type with rrf|linear fusion, verified live. (fusion: "learned" is parsed and rejected with a 400 naming the supported values — it is not implemented.) (The ES-native top-level {query, knn} body also unions both halves since rc.72 — see Known partials for its performance caveat.)
  • Zero-config folder onboarding — xerj autoindex <folder> sniffs files, infers datasets, and creates one index per dataset: tree-sitter AST extraction for 34 languages (symbols, defs, line numbers — the #295 expansion; still open: Clojure and source-SQL wait on usable grammar crates, Nim/Crystal have none, fixed-form Fortran is deliberately unclaimed), CSV/JSON/JSONL/XML/YAML/SQLite/PDF/DOCX/HTML/log/.eml/mbox (Google Takeout) formats, .gitignore/.xerjignore support, incremental re-runs, and a machine-parseable progress stream.
  • Agent-memory REST API (/_memory/*), second-brain knowledge graph (/_graph), anomaly detection (_ml with continuous datafeeds), auto-embed on ingest (default embedder is deterministic lexical feature-hashing — never described as neural; --embed-mode neural runs the in-binary BERT encoder, --embed-mode proxy an external endpoint).
  • The System One wire, answered locally (since rc.76) — POST /v1/systemone (native REST) and POST /_decide (ES-compat): typed judgement questions (noul / choice) answered by a rank-weighted kNN vote over a labelled-history index — no model, no provider key, no egress (benchmarks/decisions-as-retrieval: Banking77 0.933 accuracy / ECE 0.012, SMS spam 0.983). The acceptance gate is the unmodified pip jev-reranker ranked off a XERJ node (benchmarks/systemone-gate). It is not a judge model and claims no zero-shot judgement: where there is no history, use the rerank stage and its provider.
  • Columnar storage — the ZBS2 columnar block with 9 domain-aware encodings, ZSTD/LZ4 codecs, and SQ8 vector quantization, wired into the segment write path.
  • Bulk / scroll / delete-by-query, aliases, index templates, executed index-lifecycle policies (ISM-modeled, _ilm/* + _plugins/_ism/*, since rc.15), _cat/*, _cluster/health, _count / _msearch / _mget, _update / _update_by_query — all live-verified.
  • A single native binary, statically linked, no JVM, sub-second cold start.

The release-by-release record of how all of this landed is CHANGELOG.md — this file no longer duplicates it. The record is now complete rc.1 → rc.78: the rc.19–rc.70 gap was backfilled on 2026-09-26 from git log (merge subjects and commit bodies) and the GitHub release records, with a provenance banner in the file — the reconstructed entries are drier than ship-time ones, and every issue/PR link they carry was checked to sit inside its release window (PR #1035).

Next release — v1.0.0-rc.79

rc.78 was cut on 2026-09-29 — its full contents are the CHANGELOG.md entry, not this file. It is the index-size stage-1 and corpus-packs release: the durable-bytes epic #1038 landed its first four format changes, each gated by the size harness — .dv bit-packing with a per-column chooser (ZNV2, PR #1040), .post block framing (ZPS2, PR #1041), typed-int / copy-of stored columns with a merge-path zstd chooser (ZBS4, PR #1042), and columnar-varint .meta records (ZFM5, PR #1043) — with the committed same-day pair at 5,398,478 → 3,989,254 B durable (−26.1 %) on the 100k harness and a blog write-up (PR #1044); and xerj corpus build shipped portable, signed corpus packs (PRs #1046–#1048, publish hardening #1049–#1051) — the records half of #1030 — with the first showcase pack rust-vulns published daily from a scheduled workflow.

The same release closed the idle-cost arc — #874 met at ~3× margin behind the at-rest fixture and its CI gate (the gauge-loop deletion PR #1020, the lazy seen-set PRs #1025/#1034, idle 206 → 64 kB per index) — the flush publication bracket (#1015, PR #1018), the ingest-RSS investigation (#950, PR #1017, plus POST /{index}/_cache/clear in PR #1009), and the by-query truncation class whole (_delete_by_query #1019/PR #1021, _update_by_query #1022/PR #1023). It also carries the email-labelling measurement and field report for discussion #1012 (PRs #1026/#1027), the README's plain-words Jev explanation (PR #1010), the llms.txt and whole-surface status corrections (PRs #1033/#1037), the CHANGELOG rc.19–rc.70 backfill (PR #1035), the 2026-09-26 review of this file (PR #1036), and the site work — corner brand and blog endings (PR #1045).

In flight for rc.79: the first five items of the ten-item program to 1.0 posted in discussion #1054 ("Improvement plan till first official release", 2026-09-30) and indexed in the pinned plan tracker #1065: #1055 (xerj_map — the catalog as an MCP tool), #1056 (POST /_ask + xerj_plan — prompt in, validated DSL out), #1057 (local zero-shot decision head, tier 2 of System One), #1058 (max_tokens on every MCP search tool), and #1059 (stemming by default for profiler-marked semantic fields). The plan targeted the first and last of those at rc.78 — which was cut and published eight minutes before the discussion posted — so they slide to rc.79. The rc.78-milestone riders those replaced were closed the same day as the cut (2026-09-29), so this program is the whole of what is open.

Open defects. None — the tracker was emptied the same day as the rc.78 cut. The last two defects with code consequences, #1031 (a source file that grows during an autoindex run — an append-only log's normal behaviour — aborts the run, is misclassified as a bulk/backend failure, and the retry advice points at the wrong fix) and #1032 (three per-segment caches — stored_value_cache, dv_cache, id_pos_cache — stay unbounded until a merge retires their segment; the residual heap-per-doc retention after #1002), were closed 2026-09-29 as deferred, not fixed: both stand exactly as filed, neither is scheduled, and the repros live in the closed issues. They appear here as plain text rather than links because this is the section the release-notes gate checks against the live tracker at every cut. The tracker that established this file's review cadence (issue #298) was closed as abandoned the same day — the cadence is machine-enforced by docs_capability_lists and the review history lives in this file and the CHANGELOG, so the issue had nothing left to track. Two former trackers, #941 (zero-token direction) and #874 (idle cost), closed with fixes back in the rc.77 window; the two shipped-half epics above, #1038 and #1030, closed with the cut. Everything else the rc.78 window closed is recorded in the CHANGELOG, not here — a new defect gets a fresh issue and a milestone, and that is the tracking cadence now.

The road to v1.0.0 GA

The 1.0 bar: every public claim verified against the release binary, and every input either honoured or refused loudly. The gate list, each item an issue:

The program to the first official release (discussion #1054, 2026-09-30; filed 2026-09-30 as ten issues indexed in tracker #1065): rc.79 — xerj_map (#1055), /_ask + xerj_plan (#1056), the local zero-shot decision head (#1057), MCP max_tokens (#1058), stemming by default (#1059); rc.80 — self-judged hits (#1060), the decision cache flywheel (#1061), real _watcher detections (#1062), the calibration layer (#1063); v1.1.0 — the open xerj-decide model (#1064). Every item carries a measured gate that binds at ship time (e.g. #1056's ≥ 200-pair result-set F1 ≥ 0.9; #1060 must beat hybrid 0.699 SciFact / 0.345 NFCorpus beyond the run spread or it does not ship; #1062's wrong-and-confident rate ≤ 5 % at p ≥ 0.8), and the packaging rule for all ten is a benchmarks/<name> directory with raw results plus a blog post with the losses left in.

  • Close the accepted-and-ignored class (the #204 umbrella closed once its members carried their own tracking; PR #258 carried one pass of the sweep and is merged). Known members still open: nested inner_hits unparsed; random_sampler's ignored probability; weighted_avg returning HTTP 200 with an error buried in the aggregations body instead of a 400 (the 400 is part of #258). Retired from this list: nested score_mode, which was parsed-then-ignored until #862 made a nested query roll its matching children's scores into the parent per score_mode (rc.71).
  • Security hardening backlog — cargo-audit and fuzzing landed in CI with rc.16 (#207 closed); the deferred TLS/auth/symlink hardening items from the Phase-2 security backlog remain.
  • The mixed read-under-write p99 gap — the 4 benchmark losses out of 85 measured comparisons, all the same root cause (reads landing on the live memtable under writer pressure). Written up in demo/playbooks/MIXED_READ_UNDER_WRITE_FINDING_2026-07-08.md; the candidate fix is a visibility/parity-mode design decision, not a micro-optimisation, and it stays on the GA gate until fixed or explicitly descoped with the benchmark loss kept public.
  • Ship-or-descope every entry in Known partials below. GA does not ship with a "partial" section that reads like a feature list.
  • Close the CHANGELOG gap — closed 2026-09-26. rc.19–rc.70 shipped without entries; the 52 sections are now backfilled from git log and the release records, with an in-file provenance banner and every link checked to sit inside its release window (PR #1035, shipped in rc.78). A project whose pitch is verified numbers cannot ask users to reconstruct releases from git log — and now it does not.

The zero-token direction

Zero-token is the name for work XERJ does locally, inside the engine, so that a person or an agent spends no model tokens finding, judging, watching for or handing over an answer. Search already works that way — no model runs to answer a query. This section is the rest of that idea, as product work (long form: docs/ZERO_TOKEN_DIRECTION.md), with each item's status checked against main on 2026-09-18 (4d8dadbf, 32 commits after v1.0.0-rc.74) by reading the code named beside it. "In flight" means a pushed branch exists and nothing of it is on main; "planned" means no code exists. Tracking issue: #941.

Stage 1 — all of it merged and released in v1.0.0-rc.75 (each landed as its own PR; the status lines below name the release that carries each item):

  • Judged search — a rerank stage. Retrieve with BM25 or hybrid, then have a second stage re-judge the top N before the page is returned. On main since rc.75: the opt-in rerank block on _search, inert until an operator configures a key; a LOCAL cross-encoder provider was measured and did NOT beat the shipped hybrid (SciFact 0.7021 hybrid vs 0.7186 local-base, interval spanning zero; NFCorpus 0.3445 vs 0.3597 local-small), so it stays opt-in and unmerged. Before rc.75: the ES rescore block (query rescorer) and the hybrid query type with rrf|linear; there is no rerank stage. History: branch feat/rerank-stage — a rerank block on _search that hands the top hits to an external relevance judge, opt-in per request and inert until an operator configures a key. It is the only search-time feature that would send document text off the machine — proxy embeddings ([embedding] default_endpoint) send it at write time and the WAL tap replays writes to an external _bulk endpoint, both operator-configured and off by default (every outbound connection) — which is exactly why a local judge is on this list. The bar for a local model is a measured one. On the BEIR test splits with the opt-in neural embedder (--embed-mode neural, all-MiniLM-L6-v2, CPU) the shipped hybrid RRF query is already the best arm we have — nDCG@10 0.699–0.704 on SciFact and 0.345 on NFCorpus over three runs, against 0.657 / 0.302 for BM25 alone, 0.676 / 0.329 for vectors alone, and 0.686 / 0.332 for "BM25 top-30 re-ordered by the same bi-encoder", which is worse than hybrid on both. So re-ordering with the embedder we already ship is not worth building, and a local cross-encoder ships only if it beats 0.699 / 0.345 on the same harness, by more than the run-to-run spread (benchmarks/neural-path-triage/). Those figures are with the neural embedder; the default embedder is lexical feature hashing and was not part of this run. The hybrid figure was the only arm that did not reproduce — 0.6993, 0.7023 and 0.7044 on SciFact across three runs of unchanged indices — because tied RRF scores were ordered by a per-process hash seed; #940 closed that on 2026-09-21 (tied fused scores no longer ordered by HashMap iteration), and the spread above is the pre-fix measurement. The 0.699 / 0.345 bar stands until re-measured on the deterministic path — a gain inside a spread that no longer exists is still not a gain until you show it. The next attempt on that bar is tracked as #1060 (rc.80, plan item 4 of discussion #1054): a local self-judge scoring _p_relevant per hit — same gate, loses-does-not-ship rule included.
  • Share links and a guest reading room. Hand one person a link to one corpus or one document, read-only, without creating them an account. On main since rc.75: xerj share with scoped, expiring links, a guest reading room, and a --tunnel mode that names Cloudflare as a reader of the traffic. Before rc.75: nothing — there was no share or guest code in the tree, and reading a corpus needs an account session or an API key. History: branches feat/share-links (scoped, expiring links; a share can never name a system index) and feat/console-corpus-reader (the corpus browser and the reader the link opens) carried the work before it landed; pre-merge, their commits were marked unverified by their authors.
  • Mail ingest. Shipped in rc.75: xerj autoindex extracts .eml / MIME messages — headers, body, and attachments indexed as their own records (#921) — and, since #949, streams mbox mailboxes and extracted Google Takeout exports (archives are never opened) and writes replies_to / attachment_of edges from the mail headers. Measured on a synthetic mailbox only; ingest memory remains the practical limit — #948's unbounded memtable retention is fixed by #1002 (rc.77: variant-C retention ~677 → ~108 MB; the 300M run now fits its 8 GiB cap at 7,963.6 MiB, breaker engagements 178 → 2), and what remains is bounded per-segment cache retention until merge (issue #1032, closed as deferred-not-fixed on 2026-09-29; the defect record under Next release carries its status).
  • autoindex resilience. Fixed in rc.75 (#934): one refused dataset no longer aborts the run, an over-size catalog is split, a throttling node is waited out for 600 s, and --fresh adopts old state. Before it: one dataset the server refused aborted the whole run (#929), the --no-graph progress line is indistinguishable from a hang (#931), and xc-index.sh --fresh fails on a previously indexed corpus (#930). History: PR #934 (fix/autoindex-resilience) carried the fix. It is on this list because every later item assumes a folder can be indexed unattended.

Found while measuring stage 1, and gating it — each was a filed issue with a literal repro; all three closed with fixes that landed before the rc.77 tag (2026-09-21):

  • A declared analyzer stopped applying at flush: analysis.analyzer.default was honoured by the memtable only, a per-field analyzer was accepted and ignored, and an unknown analyzer name was accepted (#937). Fixed by PR #991 — a declared default analyzer is honoured at flush, segment query and merge; stemming can be turned on, and the accepted-and-ignored membership for the GA gate above is retired.
  • Neural ingest kept ~3.4 of 32 hardware threads busy: 6.1 documents/s on ~1,470-character abstracts through one _bulk stream, 29.5 documents/s from the same node when eight clients send concurrently, for the same total CPU (#938). Fixed by PR #995 — bounded window concurrency for neural _bulk embedding.
  • A semantic query on an index holding any multi-passage document was an exact scan that deep-copied every stored document per query: ~410 ms p50 on 5,183 documents, of which the model's forward pass is ~14 ms; the same query on a 10,003-document single-passage index is ~14 ms (#939). Fixed by PR #979 — the exact scan ranks addresses and hydrates only the winners; vector_column.rs's own header records the old cost as something it "used to" pay.

Stage 2 — planned; the status lines say what exists today, which in most cases is less than the API surface suggests:

  • Semantic detections. A standing question over incoming documents: a cheap percolate prefilter selects candidates, a typed judgment decides (a small closed set of outcomes, not free text), and an alert carries a calibrated probability — calibrated meaning measured on held-out labelled data and published with its reliability curve, or not shipped. Today: the percolate query is real — stored queries are matched against a supplied document. There is no alerting: PUT /_watcher/watch/{id} stores the body in memory, answers "condition":{"met":true}, and no scheduler ever evaluates a watch (es_compat.rs, put_watch); the console's .xerj_alert_rules and .xerj_alert_fires indices have schemas and are created at bootstrap, and no evaluator reads them (xerj-console-api/src/indices.rs). Before anything new is built, _watcher must either run watches or refuse them — it is an accepted-and-ignored surface today. Now tracked as #1062 (rc.80, plan item 7 of discussion #1054): evaluate or 501, percolate → decide → fire, ingest-time labels via /_decide, with the calibration layer of #1063 behind the published p.
  • A real object-storage backend. Done in rc.77: #965 wired the segment path to object storage — storage.backend = "s3" packs each segment family into one immutable ZBM1 bundle object with snapshot.json as the publication point, merges publish before retiring inputs, and a fresh node adopts the bucket (see Shipping today for the full statement). The client (s3.rs, real S3-compatible: Cloudflare R2, MinIO, AWS S3), the read-through segment cache, per-request cost accounting by billing class and the stop-don't-warn budget had landed in rc.75. Separately, xerj autoindex s3://bucket/prefix reads documents OUT of a bucket, which is a source, not a home.
  • A block index mode for logs. An index mode for log-shaped data that stores rows in time-partitioned columnar blocks and skips whole blocks by their min/max metadata, instead of building a per-term inverted index over every field. Today: the xerj-logs crate implements that design (columnar encoding, log-template extraction, time-range queries with block skipping, retention), is compiled into xerj-engine and xerj-server as a dependency, and is called from no non-test code. Log-shaped analytics run through the general columnar segment format and the aggregation suite. Wire it behind an explicit index setting and measure it against the general path on the same data, or remove it — the Log-analytics data path theme below is this same item.
  • User-code ingest plugins. Today: the ingest pipeline's transforms are built-in native Rust plugins — rename, drop, add, JSON parse, timestamp parse, PII redaction, grok, route — and they do run on _bulk. The crate is named xerj-wasm, but the wasmtime backend is not in the tree: there is no wasmtime dependency and no wasm feature, only a note that one could be added behind the same trait. Planned: a sandboxed runtime for user-supplied transforms with fuel and memory limits and no ambient filesystem or network access. Until then the refusal rule holds: a pipeline naming a processor this build does not implement is stored as unrunnable and every ingest through it is refused, never quietly run as a shorter pipeline.
  • A corpus hub of signed, pre-indexed packs. Download a reference corpus — a standard library, a specification set — already indexed, and mount it, instead of every user spending the same CPU-hours indexing the same public text. Today: the records half is built and signed — xerj corpus build (PR #1046) turns a declarative recipe (sources with licences, identity edges, merge precedence, derived fields) into a checksummed portable pack of records with a format_version readers refuse when unknown, per-file checksums verified on corpus add --from <pack>, and licence + provenance on every record; the first real pack is rust-vulns (tools/packs/rust-vulns/, identity-resolved across osv.dev and RustSec, with a test pinning its README numbers to a measured build), published by a scheduled workflow as a daily dated GitHub Release (immutable releases make tags single-use) with a detached ed25519 signature over SHA256SUMS (xerj corpus keygen/sign; corpus add --verify-sig <pubkey> checks origin before anything is indexed — the public key travels beside the recipe, never inside the pack; nobody in the prior-art survey signs their database). Still unbuilt: the hub itself (a directory of packs beyond this first one) and the pre-indexed half — the indexed-segment bundle that is the format this item would distribute, since indexing a records pack still costs each consumer the same CPU-hours. Its tracker, issue #1030, was closed 2026-09-29: the records half shipped in rc.78, the pre-indexed half is descoped until the risks below are answered, and the work re-files when it starts. Three risks decide whether the rest ships: redistribution licence (an index is a derived copy of its source text — a pack may carry only what its licence lets us redistribute, which excludes much of what people most want indexed; the builder carries licence as record data and the pack author owns the choice); pack safety (a pack is untrusted input to the segment readers, so it needs the fuzzing the on-disk formats get, and a signature proves who built a pack, not that it is safe to open); and format stability (a published pack outlives the release that built it, so the pack format needs a compatibility promise the internal segment format has never had to make).

Beyond 1.0 — themes

  • AST language expansion — 25 further tree-sitter grammars, tiered by demand, one PR per tier. #295 delivered the expansion to 34 languages and is closed; the remaining tiers have no tracking issue yet, so this theme is a plan rather than a commitment until one exists. Tier 1 (Kotlin, Swift, Scala, Dart, Lua, Perl, R, Julia, Haskell, Elixir) may land earlier in an RC if the grammar/ABI checks prove out.
  • Distributed clustering maturity — embedded Raft handles cluster metadata today, but the default run is single-node; multi-node sharding/replication hardening is a post-GA track, and XERJ does not claim multi-node production readiness until it is measured.
  • Neural embedder ergonomics — one loaded model is already shared by every index with the same embedder configuration (each index holds its own NeuralHandle, and identical configurations resolve to one lazily loaded model — shared_neural_cell in xerj-ai/src/embedder.rs; corrected 2026-09-18, the previous wording said each index held its own model). Still open: optional pre-warm at startup (the first embed pays the model load) and a larger default model option. The two measured defects this theme used to carry — ingest throughput (#938) and the multi-passage exact scan (#939) — closed with fixes (PRs #995 and #979) before the rc.77 tag.
  • Log-analytics data path — the dedicated xerj-logs columnar module is still not invoked from non-test engine/server code; log-shaped analytics run through ZBS2 + the generic aggregation suite. Wire it or remove it. The plan for wiring it is a block index mode for logs under The zero-token direction above.
  • Broader aggregation families — geo/IP/nested/join coverage beyond the current surface; the conformance suite is the measure.

Known partials

Honesty section: things that resolve without an error but do not implement full ES semantics. Each must be shipped or explicitly descoped before GA.

Re-verified against main 2026-08-30:

  • weighted_avg — not in SUPPORTED_AGG_TYPES; still returns HTTP 200 with an embedded error instead of executing or returning 400 (the 400 is part of the #258 sweep).
  • has_child / has_parent — recognised and rejected with a 400 (fail-loud by design until real parent-child join semantics exist; REJECTED_QUERY_TYPES in parser.rs).

Carried forward from the 2026-07-12 review, not re-verified live since:

  • nested — matching is real and per-element (test_nested_query) and score_mode now rolls matching children's scores into the parent (avg/max/min/sum/none, #862, rc.71). Still missing: ES's separate nested-document indexing, and inner_hits is not parsed (#204 member, above).
  • span_term / span_or / span_not — return 0 hits standalone, while composite span queries (span_near / span_first / span_containing) using the same clauses return correct hits.
  • type — mapped to MatchAll.
  • combined_fields — mapped to multi_match cross_fields; scoring is not exact. rank_feature passes through on plain fields (no rank_feature field type).
  • ES-native top-level {query, knn} — RETIRED in rc.72. It now unions both halves and scores a document reached by both as the sum, aggregations included (#825 via #879). It is kept in this list for one release with its replacement caveat, which is a performance one rather than a correctness one: the pinned clauses cannot project to the full-text index, so this shape is answered by a stored-document scan and is substantially slower than the hybrid query type (~208 ms vs ~2.5 ms measured on 100k documents at k=10). #892 carries the indexed-route fix. Correct-and-slow was chosen deliberately over fast-and-wrong.

Found something claimed but not working? That is a bug in our docs or our code — please open an issue. We would rather ship an honest roadmap than an overstated feature list.

0