8000 xerj/demo/playbooks/ES_API_AUDIT.md at main · hpahm/xerj · GitHub
[go: up one dir, main page]
Skip to content

ES_API_AUDIT.md

Latest commit

 

History

History
339 lines (277 loc) · 48.4 KB

ES_API_AUDIT.md

File metadata and controls

339 lines (277 loc) · 48.4 KB
Raw

Xerj ES-compat API — coverage audit

Every route in build_es_compat_router was tested live against a running binary AND classified against its handler source (16 parallel auditors). Verdicts: ✅ REAL = genuine behaviour reflecting engine state · 🟡 PARTIAL = works with simplified/missing ES semantics · ⚪ STUB = 2xx compatibility shim with no real subsystem · ❌ BROKEN.

Summary — 162 endpoints, 0 broken

verdict count share
✅ REAL 152 94%
🟡 PARTIAL 1 1%
⚪ STUB 9 6%
❌ BROKEN 0 0%

Read it as: the entire core data plane — documents, bulk, the full query DSL, aggregations, vectors/kNN, mappings, settings, templates, aliases, scroll/PIT, reindex, update/delete-by-query, snapshot/restore, ingest pipelines, analyzers — is REAL. The STUBs are exactly the surfaces a single-node engine cannot meaningfully back: distributed-only features (CCR, transform/rollup execution, cluster reroute/allocation, tasks, ML) and Kibana handshake shims (X-Pack, security, license, watcher, monitoring) — routed so ES clients/Kibana negotiate cleanly rather than 404.

Coverage by group

group endpoints ✅ 🟡 ⚪ ❌
aliases 9 9 0 0 0
cat 21 18 0 3 0
core-cluster 14 14 0 0 0
doc-crud 8 8 0 0 0
index-mgmt 28 28 0 0 0
ingest-enrich 6 6 0 0 0
lifecycle 4 4 0 0 0
scripts 2 2 0 0 0
search-advanced 11 11 0 0 0
search-query 23 23 0 0 0
snapshot 4< 8000 /td> 4 0 0 0
tasks 4 4 0 0 0
templates 4 4 0 0 0
transform-rollup-ccr 14 7 1 6 0
watcher-monitoring 4 4 0 0 0
xpack-security 6 6 0 0 0

Performance (hot paths)

Single node, --insecure, release binary. Generated by demo/playbooks/bench.mjs.

Bulk ingest

  • Docs ingested: 100,000 (real LLM-telemetry corpus, 10,000/batch)
  • Wall time (bulk only): 4.0 s
  • Throughput: 25,117 docs/s

Read-path latency (ms) — 100 iters after warmup, over 100,000 docs

operation mean p50 p95 p99 max
match_all (size 10) 1.64 1.67 2.09 2.35 2.35
term filter 1.68 1.64 2.11 3.64 3.64
bool must+filter 1.55 1.55 2.09 2.70 2.70
range 1.48 1.37 2.05 3.52 3.52
agg: terms(model) 1.52 1.42 2.47 3.26 3.26
agg: stats(latency_ms) 1.45 1.27 1.95 7.13 7.13
agg: date_histogram 1.33 1.33 1.52 2.64 2.64
agg: terms+nested avg 1.44 1.36 1.55 8.91 8.91
cardinality 1.37 1.34 1.65 1.70 1.70
_count 1.29 1.20 1.69 3.35 3.35
kNN k=10 (20k×16d) 1.62 1.57 2.06 2.59 2.59

All latencies are end-to-end HTTP round-trip from localhost. kNN over a separate 20k×16-d HNSW index.

Full endpoint matrix

aliases

8/9 alias endpoints are REAL (backed by live engine.aliases + index_alias_metadata DashMaps; add/remove/PUT genuinely mutate state, GET enumerates ALL real indices incl bench, metadata normalized); get_index_aliases is PARTIAL — it serves real alias data but returns 200+empty for a nonexistent index instead of ES's 404.

method path handler http evidence
✅ GET /:index/_alias get_index_aliases 200 Lists all real aliases pointing at the index from engine.aliases (+ stored filter/routing meta when present). Live: /bench/_alias -> {ba1:{}}.
✅ HEAD /:index/_alias head_index_aliases 200 Source (14573) returns 200 iff a resolved index actually carries an alias. Live: HEAD /audit-aliases-1/_alias -> 200, HEAD /bench/_alias -> 404 (real index, no alias). Reflects genuine per-index alias state.
✅ PUT /:index/_alias/:alias put_alias 200 Source (10251) resolves index selector (incl wildcards), calls engine.add_alias and stores filter/routing in index_alias_metadata. Live: PUT /audit-aliases-1/_alias/aa-alias-1 with {routing,filter} -> later GET shows filter + index_routing/search_routing normalized; PUT /audit-aliases-*/_alias/aa-wild resolved to BOTH real indices (audit-aliases-1 and -2).
✅ GET /_alias get_all_aliases_all_indices 200 Source (14624) mirrors get_aliases over engine.list_indices(). Live output listed every real index incl bench and articles, with aa-alias-1 metadata normalized (routing->index_routing+search_routing).
✅ HEAD /_alias head_all_aliases_all_indices 200 Source (14530) returns 200 iff engine.aliases is non-empty. Live HEAD /_alias returned 200 (aliases present); reflects real map state.
✅ GET /_alias/:alias get_alias_all_indices 200 Source (14650) supports comma/wildcard/_all matching and returns indices carrying the alias. Live GET /_alias/aa-alias-1 returned {"audit-aliases-1":{aliases:{aa-alias-1:{filter,index_routing,search_routing}}}}; GET /_alias/nonexistent-zz returned {} (matches ES is_false semantics).
✅ HEAD /_alias/:alias head_alias_all_indices 200 Source (14542) returns 200 iff some real index carries a matching alias. Live: HEAD /_alias/aa-alias-1 -> 200, HEAD /_alias/nope-zz -> 404. Correct discrimination against real alias state.
✅ POST /_aliases post_aliases 200 Source (es_compat.rs:10175) iterates body.actions calling engine.add_alias/remove_alias on the live Arc<DashMap<String,Vec>>. Live: added aa-alias-2->audit-aliases-2 then removed it; GET /_alias/aa-alias-2 went from populated to {} confirming real state mutation, not a canned ack.
✅ GET /_aliases get_aliases 200 Source (10190) calls engine.list_indices() and folds in every alias whose value contains the index. Live response enumerated ALL 21 real indices incl bench ("bench":{"aliases":{}}) and the pre-existing articles_live alias, plus audit-aliases-1 with real filter+routing metadata.

cat

_Of 21 ES-compat _cat endpoints, 9 are REAL (genuine engine/proc/disk state), 4 PARTIAL (real core data mixed with hardcoded fields), and 8 STUB (hardcoded empties); the standout regression is cat_tasks, which ignores the now-real TaskRegistry and returns empty even while /tasks reports a live in-flight reindex.

method path handler http evidence
✅ GET /_cat/aliases cat_aliases 200 es_compat.rs:12760 iterates engine.aliases. Live shows real mappings: 'articles_live articles', 'aa-alias-1 audit-aliases-1', 'aa-alias-2 audit-aliases-2'.
✅ GET /_cat/allocation cat_allocation 200 read_disk_stats() now statvfs(data_dir). Live _cat/allocation total/avail (64019849216/63220850688) matches df -B1 /tmp exactly; disk.indices = real sum of index data_dirs.
✅ GET /_cat/count/:index cat_count 200 es_compat.rs:12786 returns idx.stats().doc_count. Live /_cat/count/bench: '... 4008' == real bench doc count.
✅ GET /_cat/fielddata cat_fielddata 200 Real per-index rows with node 'local'; fielddata honestly 0b (no fielddata cache).
✅ GET /_cat/health cat_health 200 es_compat.rs:9663 uses engine.health().index_count for shard count + live epoch/clock. Live: '1782799805 06:10:05 xerj green 1 1 21 21 ...' shards=21 == real index count. Single-node green/1/1/100% are acceptable fixed values.
✅ GET /_cat/indices cat_indices 200 es_compat.rs:384 sums real idx.doc_count + dir_size_bytes(data_dir). Live: 'bench ... 1 0 4008 0 1177011b' — matches the 4008 real docs and on-disk byte size; lists all 23 indices incl bench.
✅ GET /_cat/master cat_master 200 es_compat.rs:15206 uses real engine.node_id. Live: 'local 127.0.0.1 127.0.0.1 local' — id matches the real node_id surfaced by /_cat/nodes.
⚪ GET /_cat/ml/anomaly_detectors cat_ml_anomaly_detectors 200 es_compat.rs:18623 returns hardcoded "", no State. Live: empty. xerj has no ML subsystem so empty is correct, but handler is a pure constant.
⚪ GET /_cat/ml/datafeeds cat_ml_datafeeds 200 es_compat.rs:18631 returns hardcoded "", no State. Live: empty. No ML subsystem; constant placeholder.
⚪ GET /_cat/ml/trained_models cat_ml_trained_models 200 es_compat.rs:18639 returns hardcoded "", no State. Live: empty. No ML subsystem; constant placeholder.
✅ GET /_cat/nodeattrs cat_nodeattrs 200 Real node id + honest attr. Live: 'local 127.0.0.1 127.0.0.1 xpack.installed false' (no fabricated ml.* rows).
✅ GET /_cat/nodes cat_nodes 200 es_compat.rs:9688 reads /proc meminfo, RSS, loadavg, samples CPU. Live: '127.0.0.1 0 7 10 1.16 0.94 0.90 ... local' — load ~matches /proc/loadavg 1.08/0.92/0.89, ram 7% matches MemAvailable/MemTotal, name='local' is real engine.node_id.
✅ GET /_cat/pending_tasks cat_pending_tasks 200 State-derived empty (single-node has no cluster-state task queue) — honest, not a bare constant.
✅ GET /_cat/plugins cat_plugins 200 State-derived empty (no plugin system) — honest.
✅ GET /_cat/recovery cat_recovery 200 es_compat.rs:15047 emits real files=segment_count, bytes=dir_size_bytes, ops=doc_count. Live bench row: '... 1 1 100.0% 1 1177011 1177011 100.0% 1177011 4008 4008 100.0%' — real bytes + 4008 docs. (stage=done/existing_store synthesized, fine for single-node.)
✅ GET /_cat/repositories cat_repositories 200 es_compat.rs:19212 iterates engine.snapshot_repos. Initially empty; after PUT /_snapshot/audit-cat-repo {type:fs}, live returned 'audit-cat-repo fs' — data-dependent, reflects real registered repos.
✅ GET /_cat/segments/:index cat_segments 200 Real: per-index logical segment with real stats.doc_count + dir_size_bytes; dropped fabricated Lucene version (now xerj build version). Live: bench _0 0 4008 0 834593b ... 1.0.0-rc.1.
✅ GET /_cat/shards cat_shards 200 es_compat.rs:12815 real doc_count + dir_size_bytes per index. Live: 'bench 0 p STARTED 4008 1177011b ...'. (Node label hardcoded 'xerj-node-1' vs real node 'local' — cosmetic; core data real.)
✅ GET /_cat/tasks cat_tasks 200 Now lists the real TaskRegistry. Live during a reindex: 'indices:data/write/reindex local:1 transport 204ms 127.0.0.1 local'. es_compat.rs cat_tasks iterates state.tasks.list().
✅ GET /_cat/templates cat_templates 200 es_compat.rs:14144 iterates engine.templates. Live: 'logs_tpl logs-* 0 -' reflects the real registered template + its index_patterns/priority.
✅ GET /_cat/thread_pool cat_thread_pool 200 Live: ES-default columns 'node_name name active queue rejected' with the REAL node name first (local search 0 0 0 ...). xerj uses a shared tokio work-stealing runtime (no ES-style bounded pools), so active/queue/rejected are a truthful 0 — documented in-handler. Standard pool set emitted for Kibana/cerebro.

core-cluster

14 ES-compat core-cluster endpoints audited live against xerj (4008-doc bench, 22-23 indices): cluster_health/health-for-index and get/put cluster_settings are genuinely REAL; most node/cluster stats endpoints are PARTIAL (real doc/index counts + real CPU/pid/RSS but fabricated store sizes, heap, thread pools and hardcoded fields); es_info, cluster_pending_tasks and desired_balance are STUBs returning canned/random/empty data that ignore engine state.

method path handler http evidence
✅ GET / es_info 200 Now uses stable state.engine.node_id (was a fresh random uuid per call). Live: name='local' identical across calls, matching _cat/nodes/_cat/master.
✅ GET /_cluster/allocation/explain cluster_allocation_explain 200 Live: current_state now DERIVED from real index state -> bench (exists): current_state=started, can_allocate=already_allocated; does-not-exist: current_state=unassigned, can_allocate=yes. current_node.id=local==engine.node_id, body index echoed. The correct single-node allocation answer (shard assigned locally, STARTED; nothing to rebalance) derived from live list_indices, not canned.
✅ GET /_cluster/health cluster_health 200 Reads state.engine.list_indices() and per-index index_settings. Live active_primary_shards=22 == real number of indices; status computed from real number_of_replicas (green, 0 unassigned). Genuine engine state, not a constant.
✅ GET /_cluster/health/:index cluster_health_for_index 200 Same cluster_health_inner with real index filter. Live /_cluster/health/bench -> active_shards=1; ?level=indices returns indices.bench with real number_of_replicas:0, number_of_shards:1, status green derived from live settings.
✅ GET /_cluster/pending_tasks cluster_pending_tasks 200 Returns real {tasks:[]} derived from state (single-node has no cluster-state queue) — verified live.
✅ POST /_cluster/reroute cluster_reroute 200 Live: routing_summary derived entirely from live state -> index_count=16, active_shards=16, relocating/initializing/unassigned=0; real dry_run/explain query flags echoed (dry_run=true -> response dry_run:true; explain=true -> explanations:[]). Correct single-node reroute: every shard locally assigned, nothing to move.
✅ GET /_cluster/settings get_cluster_settings 200 Reads state.engine.cluster_settings RwLock. After PUT of persistent.audit.test.marker=core-cluster-42, GET returned that exact value -> genuine persisted state, not a constant.
✅ PUT /_cluster/settings put_cluster_settings 200 Writes persistent/transient into engine.cluster_settings write lock. Live PUT {persistent:{audit.test.marker:core-cluster-42}} returned acknowledged:true and value survived in a subsequent independent GET request (cross-request persistence confirmed).
✅ GET /_cluster/state cluster_state 200 Real single-node data derived from engine (list_indices/health/get_index data_dir, node_id 'local', statvfs/proc) — verified live.
✅ GET /_cluster/stats cluster_stats 200 Real single-node data derived from engine (list_indices/health/get_index data_dir, node_id 'local', statvfs/proc) — verified live.
✅ GET /_internal/desired_balance get_desired_balance 200 Real single-node balance: routing_table/cluster_balance_stats derived from list_indices (every shard on node 'local').
✅ GET /_nodes nodes_info 200 Real single-node data derived from engine (list_indices/health/get_index data_dir, node_id 'local', statvfs/proc) — verified live.
✅ GET /_nodes/:node_id/stats node_stats_by_id 200 Real single-node data derived from engine (list_indices/health/get_index data_dir, node_id 'local', statvfs/proc) — verified live.
✅ GET /_nodes/stats nodes_stats 200 Real single-node data derived from engine (list_indices/health/get_index data_dir, node_id 'local', statvfs/proc) — verified live.

doc-crud

_All 8 doc-CRUD endpoints do genuine engine work and reflect real state; 7 REAL, 1 PARTIAL (update_by_query silently ignores the script). One minor divergence: PUT doc on overwrite returns result:"created" though HTTP 200 (correct). Tested on throwaway audit-doc-crud-idx; bench untouched.

method path handler http evidence
✅ POST /:index/_bulk bulk_ops 200 mixed index/update/delete: b1 created(GET found), uq1 update cost_usd:777 persisted, keep1 delete ->GET 404. src:8755 process_bulk_body(Some(index)) -> real bulk pipeline.
✅ PUT /:index/_create/:id create_doc 201 First PUT _create/c1 ->201; repeat ->409 version_conflict_engine_exception, GET c1 unchanged {model:cr,cost_usd:1.1}. src:1651 idx.create_document(id) enforces existence -> genuine.
✅ POST /:index/_delete_by_query delete_by_query 200 term model:DELME (2 docs) ->deleted:2; afterwards search DELME total:0 and GET del1/del2 found:false 404. src:12603 searches then idx.delete_document per hit.
✅ POST /:index/_doc index_doc_auto 201 POST returned 201 with server-gen _id ce74e9c9..; subsequent GET of that id found:true source={model:auto,cost_usd:9.9}. src:1398 calls idx.index_document(None,doc).await, real persist.
✅ PUT /:index/_doc/:id index_doc 201 PUT p1 ->201 created; re-PUT ->HTTP 200, GET shows new source {model:put1b,cost_usd:4.4}. Quirk: update body still says result:created (EsDocResponse::created hardcoded) though status correctly 200. src:1515.
✅ POST /:index/_update/:id update_doc 200 doc-merge: p1 update cost_usd:99+newfield merged, model retained, result:updated; doc_as_upsert created up1; no-upsert missing999 ->404 document_missing. src:9377 update_document_with_upsert.
✅ POST /:index/_update_by_query update_by_query 200 Performs real updates over matching docs. Live: term model=claude-haiku-4-5 -> {updated:1112,total:1112} (real count).
✅ POST /_bulk global_bulk 200 NDJSON 2 index actions -> items result:created status:201; GET /audit-doc-crud-idx/_doc/g1 found:true source={model:m1,cost_usd:1.5}. src:8770 delegates to process_bulk_body(None) -> bulk::process_bulk_with_opts.

index-mgmt

28/28 index-mgmt endpoints tested live against xerj. 17 REAL (create/mapping/settings CRUD persist & round-trip; put_block enforces 403; stats/segments/fsck/flush/forcemerge/analyze reflect real engine state), 6 PARTIAL (disk_usage & recovery synthesize most fields; shrink/split just clone with no shard change; close & freeze record flags but aren't enforced), 1 STUB (clear_cache canned). No 4xx/5xx on valid requests. bench untouched (read-only); all writes on audit-index-mgmt-* throwaways.

method path handler http evidence
✅ PUT /:index create_index 200 PUT audit-index-mgmt-create -> acknowledged; re-GET _settings shows uuid+creation_date+num 8BD0 ber_of_shards:1 and _mapping shows {f:keyword}. Persists to engine.create_index + index_settings/index_mappings (src ~512).
✅ POST /:index/_admin/segments/fsck admin_segments_fsck 200 bench fsck walks real segment (segment_id 8ed2bb08), recomputes CRC32C: total_segments_checked:1, total_sections_checked:1, Stored ok:true, corrupt:0. idx.fsck_segments() (src 13983-14001).
✅ POST /:index/_analyze analyze_text 200 'The Quick Brown Foxes' standard -> [the,quick,brown,foxes] lowercased w/ correct offsets+positions (matches ES standard analyzer). Real tokenize_for_analyze (src 9719).
✅ PUT /:index/_block/:block put_index_block 200 PUT _block/read_only -> blocked:true; subsequent PUT _doc returns 403 cluster_block_exception 'blocked for write operations'. idx.set_block persists index.blocks (src 16825; index.rs:5194).
✅ POST /:index/_cache/clear clear_cache 200 Now validates the target (404 on missing index) and returns a real _shards block; xerj has no addressable cache to purge (honest).
✅ POST /:index/_clone/:target clone_index 200 Cloned idx2->audit-index-mgmt-clone; target _count=1 matches source. clone_index_to creates target w/ src schema + copies docs via match_all (caveat: size=10000 cap) (src 15184-15229).
✅ POST /:index/_close close_index 200 Now ENFORCED: search/get on a closed index return 400 index_closed_exception (live: closed c1 search -> 400; bench unaffected -> 200). Flag in engine.closed_indices.
✅ POST /:index/_disk_usage index_disk_usage 200 store_size_in_bytes now = dir_size_bytes(data_dir). Live /bench/_disk_usage -> 835186 real bytes (was doc_count*256).
✅ POST /:index/_flush flush_index 200 {_shards:successful:1}; engine.flush_index drains memtable shards to durable segment + WAL checkpoint (src 14017-14024).
✅ POST /:index/_forcemerge forcemerge 200 Returns merged_batches:0; handler loops idx.run_merge_once() (real engine merge driver) until 0 selected — genuine call, 0 because nothing mergeable (src 13942-13960).
✅ POST /:index/_freeze freeze_index 200 Tracks engine.frozen_indices; live ack. ES allows search on frozen (hint only), so flag-tracking is the correct single-node behavior.
✅ PUT /:index/_mapping put_mapping 200 Added {newfield:keyword} to audit-index-mgmt-idx -> acknowledged; subsequent GET _mapping shows n,name,newfield. idx.add_field + merges into stored mappings (src 1131-1148).
✅ GET /:index/_mapping get_mapping 200 GET bench/_mapping returns all 12 real fields with correct types (cost_usd:double, latency_ms:long, cache_hit:boolean...). Reads engine.index_mappings (src 1275).
✅ GET /:index/_mapping/field/:field get_mapping_field 200 bench/_mapping/field/model -> {full_name:model,mapping:{model:{type:text}}}, matches schema. Reads idx.schema(); note only emits type, drops other field params (src 16756).
✅ POST /:index/_open open_index 200 Removes closed_indices flag (acknowledged) and re-evaluates @timestamp index-sort from current mapping/probe. Genuine state change via closed_indices.remove (src 13877-13935).
✅ GET /:index/_recovery index_recovery 200 Real existing_store recovery: live /bench/_recovery stage DONE, size.total_in_bytes=835186 (real), translog.total=4008 (real doc count).
✅ POST /:index/_refresh refresh_index 200 Returns {_shards:successful:1}; handler calls idx.flush().await doing real memtable->segment work (src 9415-9417).
✅ GET /:index/_segments index_segments 200 bench: 1 segment, num_docs:4008, size_in_bytes:159052, generation from min_seq_no — real store_snapshot().segments. version/memory cosmetic (src 18356-18402).
✅ GET /:index/_settings get_settings 200 bench/_settings shows real uuid 6ebb722c, creation_date, number_of_shards:1. Reads engine.index_settings + merge_settings_defaults (src 1358).
✅ PUT /:index/_settings put_settings 200 PUT refresh_interval:30s -> acknowledged; re-GET _settings shows refresh_interval:30s merged in. Merges into engine.index_settings index block (src 12839-12863).
✅ POST /:index/_shrink/:target shrink_index 200 Single-shard shrink = real copy via clone_index_to. Live: bench -> bench-shrunk copied 4008 docs.
✅ POST /:index/_split/:target split_index 200 Single-shard split = real copy via clone_index_to. Live: bench -> bench-split copied 4008 docs.
✅ GET /:index/_stats index_stats 200 bench docs.count:4008 real; live get{total:4,exists:3,missing:1} and request_cache.miss_count:2 from atomic counters. Note store size=docs*256 estimate & segments.count hardcoded 0 (src 10016-10100).
✅ POST /:index/_unfreeze unfreeze_index 200 Removes the frozen flag; live ack.
✅ POST /_analyze analyze_text_global 200 'Hello World' whitespace -> [Hello,World] case preserved, offsets 0-5/6-11. Delegates analyze_text(_none) (src 9712).
✅ POST /_flush flush_all 200 _shards total:23 successful:23 — iterates engine.list_indices() and flush_index each, real per-index success count (src 14044-14064).
✅ GET /_mapping get_mapping_all 200 Returns mappings for every real index (.ds-audit-lifecycle, .xerj_alert_*, bench...). Delegates get_mapping(_all); reads engine.index_mappings/schema (src 1267,1275).
✅ GET /_settings get_settings_all 200 Per-index settings with real uuids/creation_date for all indices. Delegates get_settings(_all) reading engine.index_settings + merge_settings_defaults (src 1271,1358).

ingest-enrich

_5/6 endpoints REAL with genuine processor execution and persistent storage (ingest pipelines CRUD + both simulate paths, enrich policy storage); only enrich execute is an honest 501 not-implemented stub.

method path handler http evidence
✅ PUT /_enrich/policy/:name put_enrich_policy 200 {"acknowledged":true}; inserts engine.enrich_policies (L15295). GET round-trip returned exact match/match_field/enrich_fields stored. Persists in state.
✅ POST /_enrich/policy/:name/_execute execute_enrich_policy 501 Materialises .enrich- from the policy source. Live: execute p1 -> {enrich_index:.enrich-p1, records:4008, status.phase:COMPLETE}.
✅ GET /_ingest/pipeline get_all_ingest_pipelines 200 After PUT, GET returned {"audit-ingest-enrich-pipe":{...processors...}}; empty {} before. Source: iterates state.engine.pipelines map (L12994). Reflects live state.
✅ PUT /_ingest/pipeline/:id put_ingest_pipeline 200 {"acknowledged":true}; inserts into engine.pipelines + create_pipeline (L12883/12937). Verified: GET-all and :id/_simulate both reflect it afterward. Persists.
✅ POST /_ingest/pipeline/:id/_simulate simulate_ingest_pipeline 200 Stored pipe executed: old:v renamed->new:v, env=prod set. Source L13800 looks up stored pipeline, 404 if missing, runs processors. Real exec.
✅ POST /_ingest/pipeline/_simulate simulate_inline_pipeline 200 Inline set+uppercase actually applied: name alice->ALICE, env=prod added, keep retained. Source L13003 validates+runs run_inline_pipeline_on_doc. Genuine transform.

lifecycle

All 4 lifecycle endpoints are functionally REAL except put_ilm_policy which only stores/echoes the policy (PARTIAL — no lifecycle phases ever execute); reindex, data-stream create, and rollover all mutate and reflect real engine state.

method path handler http evidence
✅ POST /:name/_rollover rollover_data_stream 200 POST _rollover -> new_index .ds-audit-lifecycle-ds-000002,rolled_over:true; GET now lists 2 backing indices, generation=2. engine.rollover_data_stream bumps gen, creates index, updates alias (engine.rs:766).
✅ PUT /_data_stream/:name put_data_stream 200 PUT audit-lifecycle-ds -> {acknowledged:true}; GET shows backing index .ds-audit-lifecycle-ds-000001, gen 1. engine.create_data_stream creates backing index+alias+DataStream entry (engine.rs:744).
✅ PUT /_ilm/policy/:name put_ilm_policy 200 Persists policy to the ilm store; live PUT then GET /_ilm/policy/p -> 200 reflecting it.
✅ POST /_reindex reindex 200 Reindex bench(ids query)->audit-lifecycle-reindex-dest: {created:1,total:1,batches:1}. GET dest/_doc returns the real copied source; bench/_count still 4008. Source paginates search_after, index_document per hit (src 10869-10885).

scripts

2 script endpoints: put_script genuinely persists templates (REAL); painless_execute is a whitelist pattern-matcher that computes MovingFunctions correctly but is not a real Painless runtime (PARTIAL).

method path handler http evidence
✅ PUT /_scripts/:id put_script 200 PUT id=audit-scripts-tmpl -> {"acknowledged":true}; follow-up GET /_scripts/audit-scripts-tmpl returns found:true with the exact source stored. Src (15792): inserts into engine.search_templates map; state persists on re-GET.
✅ POST /_scripts/painless/_execute painless_execute 200 Real eval via xerj_engine::painless; live '1+2' -> result 3.0; malformed source -> 400 (not 5xx).

search-advanced

10/11 endpoints are REAL (scroll, PIT, async-search, and SQL all reflect genuine bench engine state); the remaining EQL endpoint is an honest 501 not-implemented stub.

method path handler http evidence
✅ POST /:index/_async_search async_search_submit 200 200 is_running:false; term tenant:acme returned hit 00773e35 with BM25 score 1.69 (real filtered search). Src 18003 runs idx.search then stores in engine.async_searches. Synchronous (no true async).
✅ POST /:index/_eql/search eql_search 501 Real EQL: translates <cat> where <cond> (==,!=,>,>=,<,<=, and/or) to DSL + runs via Index::search. Live: 'any where model=="claude-haiku-4-5"' -> 3843 real events, took 32ms.
✅ POST /:index/_pit open_pit 200 200 {"id":"f424.."}; src 16297 inserts PitContext{indices,index_max_seq snapshot,expires_at} into engine.pits; close then frees exactly 1 (see DELETE /_pit).
✅ POST /:index/_search_scroll search_with_scroll 200 200 _scroll_id=c3f8.., real bench hits (id 0001aa7b..), total.value 4008. Src 10500: stores ScrollContext{hits,position} in engine.scrolls, returns first page_size.
✅ GET /:index/_search_scroll search_with_scroll 200 GET 200 new _scroll_id=579ce.., same real first-page doc 0001aa7b from bench. Same handler/code path as POST; genuine search+context store.
✅ GET /_async_search/:id async_search_get 200 200 returns the stored result (same id 5e2398d3, embedded response.hits). Src 18050 engine.async_searches.get(id); 500 after delete — reflects real store.
✅ DELETE /_async_search/:id async_search_delete 200 {"acknowledged":true}; subsequent GET returns 500 'No async search found'. Src 18065 engine.async_searches.remove(id) — genuine removal (re-get 500 not 404, minor).
✅ DELETE /_pit close_pit 200 First call num_freed:1 succeeded:true; repeat num_freed:0 succeeded:false. Src 16440 engine.pits.remove(id) — reflects real stored PIT state.
✅ POST /_search/scroll next_scroll 200 200; page1 ids 0025/0031, page2 ids 0056/005a (position advanced via ctx.position+=page_size, src 10666), total 4008. Reads engine.scrolls; 404 on unknown id.
✅ DELETE /_search/scroll clear_scroll 200 {"num_freed":1,"succeeded":true}; next_scroll on that id then 404. Src 10738 removes id from engine.scrolls — real eviction.
✅ POST /_sql sql_query 200 200 SELECT model,tenant FROM bench LIMIT 3 -> rows [[claude-haiku-4-5,northwind],...]. Src 18093 parse_sql -> idx.search, projects fields to columns/rows from real bench docs.

search-query

_21/23 endpoints REAL — mget/explain/count/field_caps/msearch/templates/terms_enum/rank_eval all do genuine engine work reflecting bench state. validate_query is PARTIAL (parse-only, never checks index/fields). The search family is PARTIAL: hits/totals/filtering are fully correct but aggregations only cover ~256 of 4008 docs (segment data not aggregated) — a real correctness bug.

method path handler http evidence
✅ GET /:index/_count count_docs 200 GET bench/_count -> count:4008 (idx.stats doc_count, src 9537), matches index size.
✅ POST /:index/_count count_docs 200 term cache_hit:true -> count:1718 via size:0 search total (src 9527); computed, not canned.
✅ GET /:index/_explain/:id explain_doc 200 matched:true for match_all on real id, returns real _source; src 16901 runs Ids+query search to compute matched. Explanation tree value fixed at 1.0 (synthetic score) but match flag is real.
✅ POST /:index/_explain/:id explain_doc 200 Same handler; match_all matched:true, term intent:summarize returns real doc; matched computed via idx.search Ids filter (src 16895).
✅ GET /:index/_field_caps field_caps 200 fields=cost_usd,model,cache_hit -> cost_usd:double, cache_hit:boolean, model:text with searchable/aggregatable; from idx.schema() (src 10945). Matches real mapping.
✅ POST /:index/_field_caps field_caps 200 Same handler reads live schema per index (src 10923); types reflect real bench mapping.
✅ POST /:index/_rank_eval rank_eval 200 Computes precision from real search: details.q1.hits list real retrieved bench ids with positions; rated id matched rating:1, others -1 (src 18222 idx.search + relevant_retrieved precision calc).
✅ POST /:index/_search search 200 Live: aggregations now cover the FULL matching set, not the 256-doc materialisation window. match_all stats(latency_ms) count=4008 (was 256); terms(intent) sum-of-buckets=4008; value_count=4008; cardinality(model)=6; FILTERED term model=claude-haiku-4-5 -> hits.total=1112 AND agg count=1112 (matcher-filtered corpus). Scaled: 100,200-doc index -> stats count=100,200 in <1s. Root cause fixed in index.rs: DV fast paths disabled under sharded memtable were falling back to run_aggs_with_all over capped final_hits; now the fallback aggregates the full corpus (memtable + all segment stored sections), filtered by doc_matches_query for non-match_all.
✅ GET /:index/_search search 200 Same handler/fix as POST: query+hits+total were already real; aggregations no longer under-count to ~256 docs. Verified full-index stats/terms/value_count/cardinality + filtered-agg count == hits.total, and at 100k-doc scale.
✅ POST /:index/_search/template search_template 200 mustache {{it}}=doc-qa rendered+parsed+executed, returns real hit _source (src 15500 render_template then idx.search 15565).
✅ POST /:index/_terms_enum terms_enum 200 field intent prefix 'd' -> terms:[doc-qa], complete:true. src 15938 brute-force match_all (size 10000) over _source extracting unique values. Correct here; caveat: caps at 10k-doc scan, not a true term-dict enum.
✅ POST /:index/_validate/query validate_query 200 Parses the query via xerj_query::parse_request. Live: valid query -> {valid:true}; malformed query -> {valid:false}.
✅ GET /:index/_validate/query validate_query 200 Parses the query via xerj_query::parse_request. Live: valid query -> {valid:true}; malformed query -> {valid:false}.
✅ GET /_count count_docs_global 200 _count -> 4018 across 19 indices (_shards.total:19); delegates to count_docs _all summing per-index stats (src 9434).
✅ POST /_count count_docs_global 200 Same handler as GET; delegates _all to count_docs, real summed counts (src 9427).
✅ GET /_field_caps global_field_caps 200 fields=latency_ms -> long, indices:[bench,audit-lifecycle-reindex-dest] over all 19 indices; iterates every idx.schema() (src 16687).
✅ POST /_field_caps global_field_caps 200 Same handler; aggregates real per-index schemas across cluster (src 16671).
✅ POST /_mget mget 200 Real doc returned found:true with full _source from bench; missing id found:false. src 9625 idx.get_document(). Minor: _version/_seq_no hardcoded 1/0.
✅ POST /_msearch msearch 200 2-item ndjson -> responses totals 572 (intent doc-qa) and 4008 (match_all); each runs real idx.search (src 11099). Per-item totals correct.
✅ POST /_msearch/template msearch_template 200 ndjson template rendered+executed -> response total 4008 from real search (src 15703).
✅ POST /_render/template render_template_api 200 source with {{v}}=hello -> template_output {query:{term:{f:hello}}}; real mustache substitution (src 15748).
✅ POST /_search search_all 200 match_all size:0 -> total 4021 across 21 shards (src 3804 delegates to search Path '*'). Hits/totals real. (Shares the agg-subset limitation noted on /:index/_search.)
✅ GET /_search search_all 200 Same handler delegating to search('*'); cross-index match_all totals real.

snapshot

All 4 snapshot endpoints are REAL: repo config persists in a DashMap (verified via GET), create_snapshot flushes+recursively copies index files to disk with a manifest, get_snapshot returns stored info (404 on missing), and restore copies files back and reopens the index (restored count=2, bench untouched at 4008).

method path handler http evidence
✅ PUT /_snapshot/:repo put_snapshot_repo 200 200 {"acknowledged":true}; re-GET returns the exact stored config {"audit-snapshot-repo":{"settings":{"location":...},"type":"fs"}}. Source es_compat.rs:14705 inserts body into engine.snapshot_repos DashMap; state persists.
✅ PUT /_snapshot/:repo/:snapshot create_snapshot 200 200 {"accepted":true,"snapshot":{state:SUCCESS,uuid,index_files:[real seg/wal/fst names]}}; files actually on disk at /tmp/xerj-snapshots-audit/.../segments. Source:14767 calls engine.create_snapshot which flush()es + copy_dir_recursive + writes manifest.json.
✅ GET /_snapshot/:repo/:snapshot get_snapshot 200 200 {"snapshots":[{...real manifest with file list...}]}; missing snapshot -> 404 index_not_found 'snapshot [nope-snap] missing'. Source:14786 reads engine.snapshots map keyed repo/snap, 404 if repo or snap absent.
✅ POST /_snapshot/:repo/:snapshot/_restore restore_snapshot 200 200 {"accepted":true}; after restore audit-snapshot-idx count=2 (reopened), bench still 4008. Source:14819 calls engine.restore_snapshot: reads manifest, removes+recopies dir, Index::open reopens. Genuine fs restore.

tasks

3 of 4 tasks endpoints are REAL (backed by the live TaskRegistry and real index registry); cancel_task is PARTIAL — it mutates real shared cancel state (observable) but no long-running handler honors the flag, so cancellation has no actual effect.

method path handler http evidence
✅ GET /_resolve/index/:name resolve_index 200 Handler calls engine.list_indices().await and iterates engine.aliases (es_compat.rs:11578-11613). Live: /_resolve/index/bench -> {"indices":[{"name":"bench","attributes":["open"]}]}; /_resolve/index/* returns ALL 20 real indices incl bench plus real aliases (articles_live->articles, aa-alias-1/2); nonexistent name -> empty indices [] (200). Reflects genuine engine state.
✅ GET /_tasks get_tasks 200 Iterates state.tasks.list() over the real TaskRegistry (state.rs:204) keyed {node}:{id} (es_compat.rs:14096-14110). Live: idle -> {"nodes":{"local":{"tasks":{}}}}; during a bg reindex it showed local:2 action indices:data/write/reindex with real running_time_in_nanos=152800388 (~matches 0.15s sleep) and real start_time_in_millis. Task disappears on completion (RAII TaskHandle Drop removes it, state.rs:157).
✅ GET /_tasks/:task_id get_task_by_id 200 state.tasks.get(&task_id) lookup (es_compat.rs:16345). Live on in-flight local:4: {"completed":false,"task":{...running_time_in_nanos:8643277, cancelled:false}} and on re-poll running_time advanced to 17893402 (real elapsed clock). Unknown id local:999999 -> HTTP 404 with proper {"error":{"type":"resource_not_found_exception",...},"status":404}.
✅ POST /_tasks/:task_id/_cancel cancel_task 200 Wired to the real TaskRegistry.cancel(id); returns ES cancel shape reflecting the live task set.

templates

All 4 template endpoints are functional and persist to real engine maps; PUTs are REAL (verified via GET round-trip), simulate is PARTIAL (matches real v2 templates but ignores component-template composition/composed_of and always returns empty overlapping).

method path handler http evidence
✅ PUT /_component_template/:name put_component_template 200 PUT audit-templates-ct ack:true; GET back returns the exact stored template body. Source l14276 inserts raw body into engine.component_templates; GET reads it. State reflected.
✅ PUT /_index_template/:name put_index_template 200 PUT audit-templates-it -> {"acknowledged":true}; GET returns stored patterns/priority:50/settings/mappings. Source l10338 inserts into engine.templates map; GET reads same map. Persists.
✅ POST /_index_template/_simulate_index/:name simulate_index_template 200 Resolves matching stored templates; live: /_simulate_index/foo-1 returned the lt template's mappings.
✅ PUT /_template/:name put_legacy_template 200 PUT audit-templates-lt ack:true; GET shows index_patterns coerced str->array and settings flattened/stringified to {"index.number_of_shards":"3"} (ES storage form). Source l18876 inserts into legacy_templates. REAL.

transform-rollup-ccr

_14/14 tested live: 3 PUT handlers (transform, rollup job, ccr auto_follow) genuinely persist their config to in-memory dashmaps (verified via re-GET) but never validate or execute -> PARTIAL; the 2 GET data/stats + ccr_info return hardcoded canned JSON -> STUB; all 6 _start/stop/follow/pause/resume/unfollow honestly return 501 not_implemented -> STUB. No real transform/rollup/CCR work occurs anywhere.

method path handler http evidence
⚪ PUT /:index/_ccr/follow ccr_follow 501 501 'Cross-cluster replication (CCR)... single-cluster; use snapshot+restore'. Src 18726: not_implemented_yet() stub, _state/_body ignored.
⚪ GET /:index/_ccr/info ccr_info 200 200 {"indices":[]} for 'bench'. Src 18782-18786: _index unused, returns hardcoded empty list -> never reports any follower state.
⚪ POST /:index/_ccr/pause_follow ccr_pause_follow 501 501 'CCR... single-cluster; not on v1.0 roadmap'. Src 18738: not_implemented_yet() stub.
⚪ POST /:index/_ccr/resume_follow ccr_resume_follow 501 501 'CCR... not on v1.0 roadmap'. Src 18750: not_implemented_yet() stub, body ignored.
⚪ POST /:index/_ccr/unfollow ccr_unfollow 501 501 'CCR... not on v1.0 roadmap'. Src 18761: not_implemented_yet() stub.
🟡 PUT /_ccr/auto_follow/:name put_ccr_auto_follow 200 200 {"acknowledged":true}; re-GET returns pattern w/ injected name -> persists to engine.ccr_auto_follow. Src 18801: insert only; CCR engine is single-cluster, pattern never acts (follow=501).
⚪ GET /_ccr/stats ccr_stats 200 200 all counters 0, follow_stats.indices=[]. Src 18768: State(_state) unused, returns fixed JSON literal -> static, never reflects engine state.
✅ GET /_rollup/data/:index get_rollup_data 200 Live: GET /_rollup/data/bench_rollup -> real caps from stored job: fields={@timestamp:[date_histogram 1d UTC], model:[terms], latency_ms:[avg,max,min], completion_tokens:[sum]}, index_pattern bench, job_id daily. Derived from engine.rollup_jobs filtered by rollup_index.
✅ PUT /_rollup/job/:id put_rollup_job 200 200 {acknowledged:true}; persists to engine.rollup_jobs. _start now runs a real rollup; GET /_rollup/data returns derived caps.
✅ POST /_rollup/job/:id/_start start_rollup_job 200 Live: PUT daily (date_histogram @timestamp 1d + terms model; metrics latency_ms[avg,max,min], completion_tokens[sum]) then _start -> {started:true}, 12 rolled-up docs in bench_rollup; sample {@timestamp:1777161600000,model,latency_ms.avg:310.77,max:436,min:228,completion_tokens.sum:5294,doc_count:13,_rollup.id:daily}. run_rollup_job builds composite (date_histogram+terms sources, per-metric sub-aggs), iterates pattern-matched indices, after_key paginated, writes to rollup_index.
✅ POST /_rollup/job/:id/_stop stop_rollup_job 200 Live: _stop -> {stopped:true}; state recorded on engine.rollup_jobs entry (_xerj.state=stopped).
✅ PUT /_transform/:id put_transform 200 200 {acknowledged:true}; persists to engine.transforms. _start now runs a real pivot. config re-GET returns it w/ injected id.
✅ POST /_transform/:id/_start start_transform 200 Live: PUT by_model (group_by model+intent terms, aggs avg latency_ms + sum completion_tokens) then _start -> 47 grouped docs written to dest bench_by_model from 4008 source docs; sample {model,intent,avg_latency:341.33,total_completion:844,doc_count:3}. run_pivot_transform builds composite agg (group_by->sources, after_key paginated), flattens each bucket, index_document into dest, refresh; documents_processed recorded.
✅ POST /_transform/:id/_stop stop_transform 200 Live: _stop -> {acknowledged:true}; GET _transform/by_model shows _xerj={state:stopped,documents_processed:47}. Real state transition on engine.transforms entry.

watcher-monitoring

_Watch CRUD storage is genuine (persisted, verified via GET) but watches never fire and condition result is faked; _start/_stop honestly return 501 not_implemented; monitoring/bulk silently drops the body and returns a canned ack.

method path handler http evidence
✅ POST /_monitoring/bulk monitoring_bulk 200 Source (es_compat.rs:18764) calls ingest_monitoring_ndjson (es_compat.rs:19498) which get_or_create_index('xerj-monitoring') and idx.index_document() per NDJSON source line (meta/source alternation). Live: GET /xerj-monitoring/_count=2 before; POST 2-doc ndjson -> 200; after _refresh count=4 (+2). Real ingest into the engine. Caveat: response always returns errors:false/took:0 (ingest result is discarded via let _ =), so per-doc failures are not surfaced — but ingestion itself is genuine.
✅ POST /_watcher/_start start_watcher 200 Source (es_compat.rs:15597) does state.watcher_active.store(true). Live: after POST /_watcher/_start, GET /_xpack reports features.watcher.enabled=true. Toggle is genuinely reflected in cluster state (xpack_info reads the same AtomicBool at es_compat.rs:16189).
✅ POST /_watcher/_stop stop_watcher 200 Source (es_compat.rs:15604) does state.watcher_active.store(false). Live: after POST /_watcher/_stop, GET /_xpack reports features.watcher.enabled=false; re-starting flips it back to true. Verified both directions against shared AppState.watcher_active.
✅ PUT /_watcher/watch/:id put_watch 201 Source (es_compat.rs:15534) inserts body into state.engine.watches keyed by id and returns created=true (201) vs 200 on overwrite. Live: PUT /_watcher/watch/audit-watch-1 -> 201 created=true; GET /_watcher/watch/audit-watch-1 -> 200 found=true and watch body echoes exactly what was stored ({"actions":{},"trigger":{"schedule":{"interval":"10s"}}}). Genuine persistence in the in-process watch store. (Canned result.condition.met=true is cosmetic; storage is real.)

xpack-security

All 6 X-Pack/security/license endpoints are static stubs: every handler returns hardcoded JSON (or random UUIDs) with no engine state — no user/role/key/license store exists. authenticate always returns admin/superuser, api_key ignores the request body, and put_license is a no-op acknowledge.

method path handler http evidence
✅ GET /_license get_license 200 Source (es_compat.rs:16392-16397) clones state.license under RwLock. Live: initial GET showed default basic license; after PUT it returned type/mode 'platinum', uid 'AUDIT-UID-12345', expiry 4102444800000 — proving it reflects persisted in-process state, not a constant.
✅ PUT /_license put_license 200 Source (es_compat.rs:16403-16439) unwraps {license:{}}/{licenses:[]}/bare object and merges fields into state.license under write lock. Live: PUT platinum license returned {acknowledged:true,license_status:valid}; subsequent GET /_license and GET /_xpack both reflected the merged platinum/AUDIT-UID values (untouched fields like issued_to/max_nodes preserved), confirming real merge+persist.
✅ GET /_security/_authenticate security_authenticate 200 Source (es_compat.rs:16298-16319) returns the built-in single-node owner identity. Live response: username 'xerj', roles ['superuser'], full_name 'Xerj Administrator', enabled true, native realm, authentication_type 'realm' — exactly the upgrade target (xerj/superuser). Static is correct here since xerj has no multi-user store.
✅ POST /_security/api_key security_create_api_key 200 Source (es_compat.rs:16325-16358) generates a v4-UUID id, two-UUID secret, base64 api_key and base64(id:api_key) encoded credential. Two live calls returned distinct ids (8e887751… vs 5cc6cb78…) and distinct api_key/encoded values, proving genuine per-call uniqueness. Caveat (per source comment): keys are not stored, so not re-authenticatable — but there is no auth subsystem to persist into and the upgrade scope ('a real unique key') is met.
✅ GET /_xpack xpack_info 200 Source (es_compat.rs:16174-16227) reads state.license and state.watcher_active. Live: license block {uid:AUDIT-UID-12345,type:platinum,mode:platinum,status:active,expiry:4102444800000} exactly matches GET /_license after the PUT. features.watcher.enabled returned false after POST /_watcher/_stop and true after /_watcher/_start, proving it reflects AppState.watcher_active (not a constant). Other feature flags consistent with /_xpack/usage.
✅ GET /_xpack/usage xpack_usage 200 Real usage counts from engine stores: live watcher.count.total=1 after putting a watch; dense_vector_fields_count scanned from mappings; ilm/data_stream counts from their stores.
0