8000 systemone: decision cache flywheel — write-back, source, human corrections (#1061) by xerj-org · Pull Request #1075 · xerj-org/xerj · GitHub
[go: up one dir, main page]

Skip to content

systemone: decision cache flywheel — write-back, source, human corrections (#1061) - #1075

Merged
xerj-org merged 2 commits into
mainfrom
decide-flywheel
Sep 30, 2026
Merged

xerj-org merged 2 commits into
mainfrom
decide-flywheel

Conversation

@xerj-org
Copy link
Copy Markdown
Owner

Motivation

Closes #1061 (plan item 6 of 10, discussion #1054). PR #1072 gave /v1/systemone and /_decide a tier ladder — but every tier-2 answer was spent when served: nothing came back to the [decisions] index, answers carried no source field (verified in the issue), and a human correction counted exactly like any other neighbour. This wires the flywheel: every answer tiers 2+ produce becomes the history that answers the next request, and corrections weigh more than the cache.

What shipped (issue ACs, mapped 1:1)

  1. Write-back — every tier-2 (local head) answer, and every tier-3 answer when that tier exists, is cached into the configured [decisions] index with {text, label, p, source, ts}:
    • text/label ride the configured text_field/label_field, so a cached answer is retrievable and votable exactly like a seeded example; p is the probability served (rounded like the wire), source the producing tier, ts RFC 3339 millis.
    • Async by construction and never fails the request: the write is tokio::spawned after the response is complete; an index that cannot be opened or written is a warn log naming it and the served answer stands.
    • A configured-but-missing index is created by the write-back itself (get_or_create_index → index_document — the exact path PUT /{index}/_doc takes in es_compat.rs), which is how a local-tier node bootstraps its own history.
    • History-tier answers are not written — they are the index already; re-writing would double their vote. /_decide abstains are not written either (an abstain was not an answer). /v1/systemone batches all local answers into one spawned write; /_decide caches its non-abstaining local answer into the index the request named (per-request index, never a different one).
  2. source per answer — /_decide top level, /v1/systemone per-question in decisions.evidence.*.source. Same string as tier, under the flywheel's name, and the value written into the cached document — DecideTier::as_str() is all three by construction. The strictly-parsed answer objects are untouched (wire compatibility kept exactly the way systemone: local zero-shot decision head (tier 2) via candle (#1057) #1072 did: extras at the top level, pinned by test).
  3. Human corrections ≥ 2x — a correction is an ordinary history document carrying human: true, written through the existing decisions-index write path (no new endpoint). The vote aggregation (vote_neighbours) reads it and weights the neighbour decisions.human_weight× its reciprocal rank. New [decisions] human_weight setting, default 2.0 (the issue's ≥ 2x floor; 1.0 = ordinary neighbour), validated at config load (not finite or ≤ 0 refuses to boot — 0 would erase a correction, negative would invert it). Only a JSON boolean true earns the weight. The boosted weight is what /_decide shows in each neighbour's weight, so the audit surface stays checkable arithmetic. Everything else keeps the 1/rank arithmetic the published measurements used.

Config surface: DecisionsConfig gains human_weight; the leaf-settings count moves 128 → 129 and every place the count is quoted is updated (config.rs header, lib.rs, xerj.default.toml commented block, SETTINGS_BY_SECTION, EXPECTED_SETTINGS); journey_zero_config passes.

Tests (scoped cargo test -p xerj-api, both feature configurations)

  • Write-back integration (tests/decide_flywheel.rs, gated decide-local): a mixed-tier /v1/systemone request (history-supported + no-support question) writes exactly one document — the local answer's, with text/label/p == the served evidence support/source: "local"/ts — and the history answer is not double-written. The same request again is answered by history from the cached answer (evidence.d1.tier == "history"): the flywheel closes end to end. Plus: /_decide caches a non-abstaining local answer (creating the configured index) and never writes an abstain (index stays absent).
  • Source-field contract (ungated — history tier's contract, any build): source on both surfaces, and the answer objects pinned to exactly their documented fields.
  • Human-weight: pure unit test (at 1.0 the plain vote wins 1.0 vs 0.5+1/3; at the default 2.0 the corrections win) AND the flip over the real /_decide vote — same documents, same BM25 ranks, only human_weight differs (1.0 → billing, 2.0 → tech), plus the boosted weight shown per neighbour.
  • Config default test: human_weight defaults to 2.0, parses from TOML ([decisions] human_weight = 3.5), and 0/negative/NaN are refused at load.
  • Unit: write_back_doc field contract (exactly five fields), human-flag strictness ("true"/1/missing are ordinary history), tier-name stability.
  • Results: cargo test -p xerj-api — 0 failed default, 0 failed --features decide-local (75 ok result lines); cargo test -p xerj-common --lib 165 passed; journey_zero_config passes; systemone_http.rs (the pre-existing contract) 20/20 in both configurations. One pre-existing heavy update_by_query test (25k docs, server-side timeout) failed twice under co-tenant load and passes on rerun in both clean and modified trees — timing, not this diff.
  • cargo clippy -p xerj-api -p xerj-common --all-targets clean in both configurations; cargo fmt applied; release builds of xerj-api succeed in both configurations.

Honest scope — what is NOT claimed

  • The Banking77 replay gate is release-time, not claimed: after 2,000 cached hosted answers, ≥ 80 % of later traffic answered by history at ≥ 0.8 confidence with accuracy ≥ 0.97 (basis: the BM25-vote blog's 87 % clearing 0.8 at 97.9 % accuracy). There is no tier-3 yet, so no hosted answers exist to cache; the replay must be run against the trained xerj-decide artifact (release: open xerj-decide model (Apache-2.0 weights on Hugging Face) + eval card #1064) at release time.
  • The TypeSafe-terms open question stands: the plan itself flags checking the TypeSafe/Jev API terms on retaining provider outputs before shipping a tier-3 write-back. Tier 2 is local-model output, so it is not bound by that question — but the gate blocks tier 3 and is noted in docs/DECISIONS.md.
  • "Hosted spend falls every week" is the metric the replay gate would license; no such number is measured here. The local-tier fixture in tests is untrained — it proves the write-back mechanics, nothing about answer quality.
  • No trained checkpoint was involved; no accuracy/ECE/latency claim is added or changed.

Reference-code note

Peer corpora are unavailable in this sandbox (per the stated exception), so this PR cites in-repo precedent: the write path is es_compat.rs's index_doc → get_or_create_index + index_document (line ~2976); the config seam follows state.rerank / state.decide resolve-once; the wire-compat extras pattern is #1072's decisions block; the validation phrasing follows the existing validate() range checks.

Files

  • engine/crates/xerj-api/src/systemone_api.rs — DecideTier::as_str, CachedAnswer/write_back_doc/spawn_write_back, neighbour_weight/is_human_correction in the vote, source on both surfaces, write-back from both handlers.
  • engine/crates/xerj-api/tests/decide_flywheel.rs (new) — the contracts above.
  • engine/crates/xerj-common/src/config.rs, lib.rs, engine/xerj.default.toml, engine/crates/xerj-engine/tests/product_experience.rs — human_weight + the 129 count.
  • docs/DECISIONS.md — the flywheel and corrections documented, with the two release-time gates named.

…tions (#1061)

Plan item 6 of 10 (discussion #1054). Three contracts, one mechanism: the
answers the decide ladder produces become the history that answers the next
request, and the corrections that steer them weigh more than the cache.

Write-back (#1061 AC 1). Every tier-2 (local head) answer — and every tier-3
answer when that tier exists — is cached into the configured [decisions]
index as an ordinary document: the configured text_field/label_field (so a
cached answer is retrievable and votable exactly like a seeded example) plus
the issue's fixed p / source / ts. The write is spawned, never awaited by
the request, and cannot fail it: an index that cannot be opened or written
is a warn log naming it, and the served answer stands. A configured-but-
missing index is created by the write-back itself (get_or_create_index, the
same path PUT /{index}/_doc takes) — that is how a local-tier node
bootstraps its own history. NOT written: history-tier answers (they are the
index already; re-writing would double their vote) and /_decide abstains
(an abstain was not an answer). /v1/systemone caches every local answer in
one spawned batch; /_decide caches its non-abstaining local answer into the
index the request named.

Source (#1061 AC 2). Every answer names the tier that produced it: /_decide
top level, /v1/systemone per-question in decisions.evidence.*.source — the
same string as tier, under the flywheel's name, and the value written into
the cached document (DecideTier::as_str is all three by construction). The
strictly-parsed answer objects are untouched, the wire compatibility #1072
established: extras ride at the top level where every verified client
ignores them (pinned by test).

Human corrections (#1061 AC 3). A correction is an ordinary history document
carrying human: true — the existing decisions-index write path, no new
endpoint. In the vote aggregation (vote_neighbours) a correction weighs
decisions.human_weight x its reciprocal rank; everything else keeps the 1/rank
arithmetic the published measurements used. New [decisions] human_weight
setting, default 2.0 (the issue's >= 2x floor), range-checked at config load
(not finite or <= 0 refuses to boot: 0 would erase a correction from the
vote, a negative value would invert it). Only a JSON boolean true earns the
weight; the boosted weight is what /_decide shows in each neighbour's
weight, so the audit surface stays honest arithmetic.

Config count: 128 -> 129 leaf settings; every place the count is quoted
(config.rs header, lib.rs, xerj.default.toml, SETTINGS_BY_SECTION,
EXPECTED_SETTINGS) updated, and journey_zero_config passes.

Tests. New engine/crates/xerj-api/tests/decide_flywheel.rs — ungated (the
history tier's contract, any build): source on both surfaces with the answer
objects pinned unchanged; the 2x flip over the real /_decide vote (same
documents, same ranks, only human_weight differs: 1.0 -> billing, 2.0 ->
tech); the boosted weight shown per neighbour. Gated on decide-local: the
full flywheel (mixed-tier request writes exactly ONE document — the local
answer's, with text/label/p-equals-served-support/source/ts — the history
answer is not double-written, and the SAME request again is answered by
history from the cached answer); /_decide caches a non-abstaining local
answer (creating the configured index) and never caches an abstain. Unit:
nei
8000
ghbour_weight 2x-flip arithmetic, write_back_doc field contract (exactly
five fields, p rounded like the wire), human-flag strictness, human_weight
default + TOML parse + range rejection, tier-name stability.

Numbers NOT claimed: the issue's Banking77 replay gate (2,000 cached hosted
answers -> >= 80% history-answered at >= 0.8 confidence, >= 0.97 accuracy)
is a release-time measurement, and the TypeSafe-terms question on retaining
provider outputs must be answered before a tier-3 write-back ships — both
noted in docs/DECISIONS.md. Hosted-spend-falls is the metric the replay
gate would license, not a result here.

cargo test -p xerj-api: 0 failed both configurations (one pre-existing
load-flaky heavy update_by_query test failed twice under co-tenant load and
passes on rerun in both trees). clippy clean both configurations; fmt
applied; release builds of xerj-api succeed in both configurations.

Reference-code note: peer corpora unavailable in this sandbox; in-repo
precedent followed (the write path is es_compat.rs's index_doc ->
get_or_create_index + index_document; the resolve-once config seam is
state.rerank/decide; wire-compat extras pattern is #1072's).
@cla-bot cla-bot Bot added the cla-signed label Sep 30, 2026
@xerj-org
xerj-org merged commit 9122252 into main Sep 30, 2026
22 checks passed
neoneye pushed a commit to agent-memory-atlas-archive/xerj-org--xerj that referenced this pull request Oct 1, 2026
The packaging half of the ten-item program's rule (discussion xerj-org#1054): a
benchmarks/<name> directory with raw results plus a blog post with the
losses left in. The gate directories landed through xerj-org#1079, xerj-org#1085, xerj-org#1086,
xerj-org#1087, xerj-org#1088 and xerj-org#1090; this is the post.

Structure follows the Jev posts (verdict first, wins then losses with
readings, then what the next window does about each loss). Every number is
verbatim from the committed gate artifacts:

- PASS: /_ask macro-F1 0.9975 over 230 pairs, 0 invalid DSL, 3-run
  determinism, wall-clock p50 1.129 ms vs the 300 ms bar (xerj-org#1076, xerj-org#1079);
  xerj_map unknown-field 400s 20/30 guessed -> 0/30 map-informed, 100%
  vs the >=80% bar (xerj-org#1074, xerj-org#1086); calibration FiQA pair-level ECE
  0.3109 raw -> 0.0088 held-out vs the <=0.10 bar, temperature 0.3163
  published as the failing comparison arm, $0.2125 of provider calls
  over 648 queries (xerj-org#1080, xerj-org#1087).
- FAIL, published as such: the lexical judge loses to its own first
  stage everywhere measured (0.7045->0.5967, 0.3419->0.2924,
  0.2382->0.1650 vs the 0.30 FiQA bar) and stays opt-in with no quality
  claim per its own rule (xerj-org#1077, xerj-org#1090); the flywheel freezes as
  shipped (12 of 2,000 cached, xerj-org#1094) and forces 19.72% coverage at
  0.9953 accuracy vs the 80%/0.97 bars (xerj-org#1075, xerj-org#1088); the tier-2
  decide head misses SMS 0.9193/0.287 vs 0.95/0.05 and AG News 0.2599
  vs 0.85, latency 8.4 ms PASS, ships behind the flag with the numbers
  published (xerj-org#1072/xerj-org#1073, xerj-org#1088).
- SPLIT: stemming +0.0160/+0.0179 BM25 nDCG@10 PASS, zero-hit 25->15
  vs <=10 FAIL — the 15 have no stem anywhere in the corpus (xerj-org#1070,
  xerj-org#1085).
- The gate that caught the stock release binary built without the
  decide-local cargo feature (flag a no-op, stock-binary-noop.txt) is
  told with the fix queued ahead of the cut, not claimed as done.
- Honest gaps stated, not smoothed: the /_ask token arm designed not
  run, the _watcher detection gate not yet measured, no HF URL for
  xerj-decide-v1. Four issues filed from the measurements: xerj-org#1091-xerj-org#1094.

Mechanics: hand-written page at landing/blog/the-rc80-gates.html,
registered in scripts/seo/pagedata.py (published/updated 2026-09-30),
card prepended to the blog hub (hub dateModified bumped), head and
footer owned by fix_heads/fix_links, og falls back to the site card.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

systemone: decision cache flywheel — every answer written back, corrections weighted ≥2x

1 participant

0