This page is a source map for humans and AI agents comparing XERJ with Lucene.
It answers six storage and search questions a Lucene, Solr, or Elasticsearch
engineer is likely to ask, then links each answer to its implementation or
official documentation. XERJ is a from-scratch Rust engine whose primary
workflow is autoindex <folder> -> autoindex map -> query; Lucene 10.3.1 is an
embeddable Java search library. Elasticsearch-wire compatibility is an adoption
bridge, not evidence that the two engines share an implementation.
This comparison is current as of 2026-08-17, except section 5's
_update_by_query note, which is updated to 2026-09-26 and pinned to that
change's merge commit. XERJ claims and source links
are pinned to commit
24711999dd866ceec2a6e7c91d934c2c27d7066c;
Lucene links point to official Apache Lucene 10.3.1 documentation. The XERJ
source wins if a nearby overview page says something different.
The default XERJ embedding fallback is lexical feature-hashing, not neural
semantic understanding (Embedder).
The built-in neural BERT backend is opt-in via --embed-mode neural and
downloads its model on first use (model loader).
| Axis | Lucene 10.3.1 | XERJ at 24711999 |
|---|---|---|
| 1. Segment and merge model | IndexWriter, immutable per-segment lifecycle, merge policy and scheduler |
WAL, sharded buffers, immutable .seg containers and sidecars |
| 2. Postings layout | BlockTree terms, packed blocks, two-level impact skips | FST/metadata terms, 128-doc PFOR blocks, no persisted skip table |
| 3. Values and ranges | DocValues for columns; BKD points for numeric and geo ranges | Numeric/keyword .dv columns with a sorted range index |
| 4. Analysis chain | Broad analysis modules and composable CustomAnalyzer |
Curated registry and shape-specific query projection |
| 5. Update and delete semantics | Delete/add documents plus restricted DocValues updates | Source overlay, logical replacement, versions and tombstones |
| 6. Vector index placement | Per-segment HNSW, filter-aware search and quantized codecs | Index-level HNSW for eligible plain queries; exact fallback elsewhere |
Lucene buffers indexing work in IndexWriter, flushes immutable segments, and
lets a MergePolicy select groups of segments for replacement by a merged
segment. Its default merge scheduler can run those merges in background
threads. See the Lucene 10.3.1 IndexWriter,
merge policy,
and concurrent merge scheduler.
XERJ has a different publication boundary. A WAL records mutations; production
Index::flush
fans out one
do_flush_shard
task per FTS shard. Each shard drains and sequence-orders its entries, builds
FTS and .dv sidecars, then calls
finalize_flush_with_publisher
to write an immutable .seg and atomically publish an ArcSwap snapshot.
The live ingest buffer is a configurable
ShardedFtsMemtable,
not one process-wide FTS lock. Its shards drain and re-sort by WAL sequence. The
background
run_merge_once
writes a replacement segment from surviving documents, and
apply_merge_with_repoints
publishes it before
retire_segment_files
retires the inputs. A XERJ segment is a checksummed container whose
SegmentWriter::finish
writes named sections
(SectionType);
FTS and doc values are separate sidecars, while stored source is a segment
section.
Lucene 10.3.1's postings format uses a BlockTree term dictionary and packed
integer blocks (128 integers in the packed path), with skip data interleaved at
two levels. The skip data carries competitive impact metadata that lets scorers
safely skip score calculation for uncompetitive documents rather than merely
advancing through every posting. See
Lucene103PostingsFormat
and its
Lucene103PostingsWriter.
XERJ's
FtsIndexWriter
writes one sidecar family per indexed field: an FST term dictionary, a
compressed .post blob, binary .meta term statistics, and encoded .norms.
The FST maps a term to a fixed metadata record; it does not point directly to a
Lucene segment file. The format and reader are documented in
FtsIndexReader.
The current encoding is defined by the
postings module format notes,
PostingsWriter::encode_term,
and its
packed-block helpers.
Doc IDs are delta-encoded in 128-doc PFOR blocks; positioned fields add packed
term frequencies and variable-byte positions, while exact fields can omit
both. Residual postings use variable-byte encoding. The shared block size is
not Lucene codec compatibility. XERJ's SkipEntry is currently in-memory
scaffolding and is not serialized, so
advance_to
scans linearly; it does not provide Lucene's persisted impact-skip behavior.
Lucene exposes typed, column-oriented DocValues alongside postings; its codec
chooses on-disk representations for numeric, so
6E24
rted, sorted-set, and related
types. The 10.3.1 reference describes .dvd and .dvm files in
Lucene90DocValuesFormat,
while
SortedSetDocValuesField
is useful for a multi-valued sorted field.
DocValues are not Lucene's primary numeric-range index. Numeric and geo points
such as LongPoint
are queried through PointRangeQuery
and stored by a BKD points format such as
Lucene90PointsFormat,
which can prune ranges of the point tree. XERJ does not claim a byte-compatible
DocValues codec or a BKD counterpart.
XERJ's closest DocValues equivalent is a per-segment {segment_id}.dv sidecar.
The
Column enum
currently has numeric and keyword columns, each with a missing-value bitmap;
keyword columns use sorted terms plus ordinals, and numeric columns retain a
sorted (value, doc_id) index for range work. The binary envelope is produced
by
encode_columns
and loaded through the engine's .dv cache.
The columns feed XERJ sorting, filters, ranges, and aggregation fast paths
without reparsing _source. They are an analogous column store, not a drop-in
Lucene DocValues codec: field eligibility, array handling, compression, and the
public mapping contract are XERJ behavior. The segment's stored source remains
the fallback and hydration authority.
Lucene's
Analyzer
creates a reusable token stream for a field. Lucene also ships broad
analysis-common
language and token-processing support, and
CustomAnalyzer
composes char filters, a tokenizer, and token filters. Analyzer output is part
of the index contract: changing analysis changes the terms in postings.
Lucene's no-argument
StandardAnalyzer
uses an empty stop-word set, lowercases terms, and does not stem. That default
is roughly equivalent to XERJ's standard preset; the material difference is
Lucene's wider module set and custom composition.
XERJ models the same conceptual stages in
AnalyzerPipeline::analyze:
char filters, one tokenizer, then ordered token filters.
AnalyzerRegistry
starts with built-ins; supported custom settings are validated and installed at
index creation
(create-time gate).
The FTS mapping builder chooses standard for text and keyword for other
mapped field types
(build_fts_field_configs);
its standard preset is Unicode word splitting plus lowercase, with no
stop-word removal or stemming.
Query analysis is shape-specific, not general index/query custom-analyzer
symmetry: match uses the requested built-in analyzer or standard when
omitted; unknown or custom names decline to stored evaluation in this default
registry
(match projection).
Keyword matches use whole-value terms. On keyword or exact fields,
match_phrase uses the whole-value term path. On analyzed text fields, its FTS
phrase path requires standard and zero slop; other analyzed shapes fall back
to stored evaluation
(match_phrase implementation).
Lucene documents are immutable once in a segment. In Lucene's own API,
updateDocument is expressed as delete-then-add; buffered deletes and live
document state are reconciled as segments are flushed and merged. The core
lifecycle is stable, but concrete segment, live-docs, and DocValues files are
codec/package details that can vary by Lucene commit; see the official
10.3.1 index package
and IndexWriter,
not universal filenames. Live-doc encoding is defined by the
LiveDocsFormat contract.
Lucene core accepts IndexableField objects, not JSON. Its
updateNumericDocValue and updateBinaryDocValue APIs only update existing
fields that were indexed with the corresponding DocValues type and DocValues
only; they cannot add arbitrary fields and are not a general JSON partial-update
API.
XERJ also does not edit an already-published segment in place. A document
update appends a new WAL/index version under the same _id; the
VersionMap
tracks the newest sequence number and whether the current version is deleted.
Old physical copies are ghosts until merge filtering removes them. Deletes are
WAL tombstones and can be carried in a segment's sequence-aware tombstone
section; the latest-version checks in merge and recovery keep an older copy
from reappearing.
The ES-shaped partial-update surface is real:
update_document
reads the current source, overlays top-level fields, and reindexes the merged
document. The API exposes /_update/{id} in
update_doc
and /_update_by_query in
update_by_query.
Since PR #1023 (merged 2026-09-26)
that handler pages the match set: an _id-keyset search_after cursor in
scroll_size batches (default 1000), a single-pass matching_ids_sorted arm
for match_all/ids selectors, ES max_docs/scroll_size honored, and a
10,000,000 max_total backstop. One request is now a full-corpus traversal;
at the 2026-08-17 pin above it hard-capped its fetch at 10,000 hits
(size: 10000). Updates are logical replacement
plus versioning, not byte-level mutation. A delete also tombstones the old HNSW
node when one exists, so a vector result cannot outlive the document
(delete_document_versioned).
Lucene treats a vector field as a first-class per-segment vector value and
delegates nearest-neighbor search to a KnnVectorsFormat reader. A
KnnFloatVectorField
creates that field, while
KnnFloatVectorQuery
executes it; the codec boundary is
KnnVectorsFormat.
For filtered kNN, Lucene can traverse HNSW subject to the filter and
automatically falls back to exact search when the filter is cheaper or graph
search reaches its visit limit; see
FilteredHnswGraphSearcher.
Lucene 10.3.1 also provides
Lucene99HnswScalarQuantizedVectorsFormat
for 4- or 7-bit scalar quantization and
Lucene102HnswBinaryQuantizedVectorsFormat
for binary quantization. These are material representation and footprint
choices, not just different graph placement.
XERJ keeps vectors out of the lexical FTS term dictionary. A mapped
dense_vector is stored in _source for hydration and maintained by
index_vectors
in an index-level persisted HNSW artifact (hnsw/;
save_hnsw_to_disk).
Graph serialization is
HnswIndex::save_to,
and the doc-ID maps are persisted alongside it. That graph is not a postings
list and is not an active FTS field sidecar.
Unfiltered top-level cosine kNN may use HNSW only after field-identity,
coverage, freshness, and size checks; candidates are fetched and exact
rescored. Exact rescoring does not make candidate selection exact: HNSW remains
approximate, so candidate-set recall can differ from brute force. Filters,
nested or passage shapes, unsupported metrics, small or stale graphs, and any
failed eligibility check use XERJ's exact brute-force path. The
kNN dispatch
and
run_knn_hnsw
make those dispatch and fallback rules explicit.
Non-claims: this page does not claim byte-compatible indexes, identical merge
policies, Lucene codec compatibility, complete analyzer parity, identical
vector recall or latency, or that an Elasticsearch API name implies Lucene
internals. It also does not compare Lucene's
facets,
near-real-time reader management through
DirectoryReader
and SearcherManager,
or its
backward-codecs
compatibility surface. This is an orientation guide pinned to the source
snapshot above, not a complete feature-parity claim.