Repository navigation
systemone: decision cache flywheel — write-back, source, human corrections (#1061) - #1075
Merged
Merged
Conversation
…tions (#1061) Plan item 6 of 10 (discussion #1054). Three contracts, one mechanism: the answers the decide ladder produces become the history that answers the next request, and the corrections that steer them weigh more than the cache. Write-back (#1061 AC 1). Every tier-2 (local head) answer — and every tier-3 answer when that tier exists — is cached into the configured [decisions] index as an ordinary document: the configured text_field/label_field (so a cached answer is retrievable and votable exactly like a seeded example) plus the issue's fixed p / source / ts. The write is spawned, never awaited by the request, and cannot fail it: an index that cannot be opened or written is a warn log naming it, and the served answer stands. A configured-but- missing index is created by the write-back itself (get_or_create_index, the same path PUT /{index}/_doc takes) — that is how a local-tier node bootstraps its own history. NOT written: history-tier answers (they are the index already; re-writing would double their vote) and /_decide abstains (an abstain was not an answer). /v1/systemone caches every local answer in one spawned batch; /_decide caches its non-abstaining local answer into the index the request named. Source (#1061 AC 2). Every answer names the tier that produced it: /_decide top level, /v1/systemone per-question in decisions.evidence.*.source — the same string as tier, under the flywheel's name, and the value written into the cached document (DecideTier::as_str is all three by construction). The strictly-parsed answer objects are untouched, the wire compatibility #1072 established: extras ride at the top level where every verified client ignores them (pinned by test). Human corrections (#1061 AC 3). A correction is an ordinary history document carrying human: true — the existing decisions-index write path, no new endpoint. In the vote aggregation (vote_neighbours) a correction weighs decisions.human_weight x its reciprocal rank; everything else keeps the 1/rank arithmetic the published measurements used. New [decisions] human_weight setting, default 2.0 (the issue's >= 2x floor), range-checked at config load (not finite or <= 0 refuses to boot: 0 would erase a correction from the vote, a negative value would invert it). Only a JSON boolean true earns the weight; the boosted weight is what /_decide shows in each neighbour's weight, so the audit surface stays honest arithmetic. Config count: 128 -> 129 leaf settings; every place the count is quoted (config.rs header, lib.rs, xerj.default.toml, SETTINGS_BY_SECTION, EXPECTED_SETTINGS) updated, and journey_zero_config passes. Tests. New engine/crates/xerj-api/tests/decide_flywheel.rs — ungated (the history tier's contract, any build): source on both surfaces with the answer objects pinned unchanged; the 2x flip over the real /_decide vote (same documents, same ranks, only human_weight differs: 1.0 -> billing, 2.0 -> tech); the boosted weight shown per neighbour. Gated on decide-local: the full flywheel (mixed-tier request writes exactly ONE document — the local answer's, with text/label/p-equals-served-support/source/ts — the history answer is not double-written, and the SAME request again is answered by history from the cached answer); /_decide caches a non-abstaining local answer (creating the configured index) and never caches an abstain. Unit: nei 8000 ghbour_weight 2x-flip arithmetic, write_back_doc field contract (exactly five fields, p rounded like the wire), human-flag strictness, human_weight default + TOML parse + range rejection, tier-name stability. Numbers NOT claimed: the issue's Banking77 replay gate (2,000 cached hosted answers -> >= 80% history-answered at >= 0.8 confidence, >= 0.97 accuracy) is a release-time measurement, and the TypeSafe-terms question on retaining provider outputs must be answered before a tier-3 write-back ships — both noted in docs/DECISIONS.md. Hosted-spend-falls is the metric the replay gate would license, not a result here. cargo test -p xerj-api: 0 failed both configurations (one pre-existing load-flaky heavy update_by_query test failed twice under co-tenant load and passes on rerun in both trees). clippy clean both configurations; fmt applied; release builds of xerj-api succeed in both configurations. Reference-code note: peer corpora unavailable in this sandbox; in-repo precedent followed (the write path is es_compat.rs's index_doc -> get_or_create_index + index_document; the resolve-once config seam is state.rerank/decide; wire-compat extras pattern is #1072's).
This was referenced Sep 30, 2026
Closed
neoneye
pushed a commit
to agent-memory-atlas-archive/xerj-org--xerj
that referenced
this pull request
Oct 1, 2026
The packaging half of the ten-item program's rule (discussion xerj-org#1054): a benchmarks/<name> directory with raw results plus a blog post with the losses left in. The gate directories landed through xerj-org#1079, xerj-org#1085, xerj-org#1086, xerj-org#1087, xerj-org#1088 and xerj-org#1090; this is the post. Structure follows the Jev posts (verdict first, wins then losses with readings, then what the next window does about each loss). Every number is verbatim from the committed gate artifacts: - PASS: /_ask macro-F1 0.9975 over 230 pairs, 0 invalid DSL, 3-run determinism, wall-clock p50 1.129 ms vs the 300 ms bar (xerj-org#1076, xerj-org#1079); xerj_map unknown-field 400s 20/30 guessed -> 0/30 map-informed, 100% vs the >=80% bar (xerj-org#1074, xerj-org#1086); calibration FiQA pair-level ECE 0.3109 raw -> 0.0088 held-out vs the <=0.10 bar, temperature 0.3163 published as the failing comparison arm, $0.2125 of provider calls over 648 queries (xerj-org#1080, xerj-org#1087). - FAIL, published as such: the lexical judge loses to its own first stage everywhere measured (0.7045->0.5967, 0.3419->0.2924, 0.2382->0.1650 vs the 0.30 FiQA bar) and stays opt-in with no quality claim per its own rule (xerj-org#1077, xerj-org#1090); the flywheel freezes as shipped (12 of 2,000 cached, xerj-org#1094) and forces 19.72% coverage at 0.9953 accuracy vs the 80%/0.97 bars (xerj-org#1075, xerj-org#1088); the tier-2 decide head misses SMS 0.9193/0.287 vs 0.95/0.05 and AG News 0.2599 vs 0.85, latency 8.4 ms PASS, ships behind the flag with the numbers published (xerj-org#1072/xerj-org#1073, xerj-org#1088). - SPLIT: stemming +0.0160/+0.0179 BM25 nDCG@10 PASS, zero-hit 25->15 vs <=10 FAIL — the 15 have no stem anywhere in the corpus (xerj-org#1070, xerj-org#1085). - The gate that caught the stock release binary built without the decide-local cargo feature (flag a no-op, stock-binary-noop.txt) is told with the fix queued ahead of the cut, not claimed as done. - Honest gaps stated, not smoothed: the /_ask token arm designed not run, the _watcher detection gate not yet measured, no HF URL for xerj-decide-v1. Four issues filed from the measurements: xerj-org#1091-xerj-org#1094. Mechanics: hand-written page at landing/blog/the-rc80-gates.html, registered in scripts/seo/pagedata.py (published/updated 2026-09-30), card prepended to the blog hub (hub dateModified bumped), head and footer owned by fix_heads/fix_links, og falls back to the site card.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Closes #1061 (plan item 6 of 10, discussion #1054). PR #1072 gave
/v1/systemoneand/_decidea tier ladder — but every tier-2 answer was spent when served: nothing came back to the[decisions]index, answers carried nosourcefield (verified in the issue), and a human correction counted exactly like any other neighbour. This wires the flywheel: every answer tiers 2+ produce becomes the history that answers the next request, and corrections weigh more than the cache.What shipped (issue ACs, mapped 1:1)
[decisions]index with{text, label, p, source, ts}:text_field/label_field, so a cached answer is retrievable and votable exactly like a seeded example;pis the probability served (rounded like the wire),sourcethe producing tier,tsRFC 3339 millis.tokio::spawned after the response is complete; an index that cannot be opened or written is awarnlog naming it and the served answer stands.get_or_create_index→index_document— the exact pathPUT /{index}/_doctakes ines_compat.rs), which is how a local-tier node bootstraps its own history./_decideabstains are not written either (an abstain was not an answer)./v1/systemonebatches all local answers into one spawned write;/_decidecaches its non-abstaining local answer into the index the request named (per-request index, never a different one).sourceper answer —/_decidetop level,/v1/systemoneper-question indecisions.evidence.*.source. Same string astier, under the flywheel's name, and the value written into the cached document —DecideTier::as_str()is all three by construction. The strictly-parsed answer objects are untouched (wire compatibility kept exactly the way systemone: local zero-shot decision head (tier 2) via candle (#1057) #1072 did: extras at the top level, pinned by test).human: true, written through the existing decisions-index write path (no new endpoint). The vote aggregation (vote_neighbours) reads it and weights the neighbourdecisions.human_weight× its reciprocal rank. New[decisions] human_weightsetting, default 2.0 (the issue's ≥ 2x floor; 1.0 = ordinary neighbour), validated at config load (not finite or ≤ 0 refuses to boot — 0 would erase a correction, negative would invert it). Only a JSON booleantrueearns the weight. The boosted weight is what/_decideshows in each neighbour'sweight, so the audit surface stays checkable arithmetic. Everything else keeps the 1/rank arithmetic the published measurements used.Config surface:
DecisionsConfiggainshuman_weight; the leaf-settings count moves 128 → 129 and every place the count is quoted is updated (config.rsheader,lib.rs,xerj.default.tomlcommented block,SETTINGS_BY_SECTION,EXPECTED_SETTINGS);journey_zero_configpasses.Tests (scoped
cargo test -p xerj-api, both feature configurations)tests/decide_flywheel.rs, gateddecide-local): a mixed-tier/v1/systemonerequest (history-supported + no-support question) writes exactly one document — the local answer's, with text/label/p== the served evidence support/source: "local"/ts— and the history answer is not double-written. The same request again is answered by history from the cached answer (evidence.d1.tier == "history"): the flywheel closes end to end. Plus:/_decidecaches a non-abstaining local answer (creating the configured index) and never writes an abstain (index stays absent).sourceon both surfaces, and the answer objects pinned to exactly their documented fields./_decidevote — same documents, same BM25 ranks, onlyhuman_weightdiffers (1.0 →billing, 2.0 →tech), plus the boosted weight shown per neighbour.human_weightdefaults to 2.0, parses from TOML ([decisions] human_weight = 3.5), and 0/negative/NaN are refused at load.write_back_docfield contract (exactly five fields), human-flag strictness ("true"/1/missing are ordinary history), tier-name stability.cargo test -p xerj-api— 0 failed default, 0 failed--features decide-local(75 ok result lines);cargo test -p xerj-common --lib165 passed;journey_zero_configpasses;systemone_http.rs(the pre-existing contract) 20/20 in both configurations. One pre-existing heavyupdate_by_querytest (25k docs, server-side timeout) failed twice under co-tenant load and passes on rerun in both clean and modified trees — timing, not this diff.cargo clippy -p xerj-api -p xerj-common --all-targetsclean in both configurations;cargo fmtapplied; release builds ofxerj-apisucceed in both configurations.Honest scope — what is NOT claimed
xerj-decideartifact (release: openxerj-decidemodel (Apache-2.0 weights on Hugging Face) + eval card #1064) at release time.docs/DECISIONS.md.Reference-code note
Peer corpora are unavailable in this sandbox (per the stated exception), so this PR cites in-repo precedent: the write path is
es_compat.rs'sindex_doc→get_or_create_index+index_document(line ~2976); the config seam followsstate.rerank/state.decideresolve-once; the wire-compat extras pattern is #1072'sdecisionsblock; the validation phrasing follows the existingvalidate()range checks.Files
engine/crates/xerj-api/src/systemone_api.rs—DecideTier::as_str,CachedAnswer/write_back_doc/spawn_write_back,neighbour_weight/is_human_correctionin the vote,sourceon both surfaces, write-back from both handlers.engine/crates/xerj-api/tests/decide_flywheel.rs(new) — the contracts above.engine/crates/xerj-common/src/config.rs,lib.rs,engine/xerj.default.toml,engine/crates/xerj-engine/tests/product_experience.rs—human_weight+ the 129 count.docs/DECISIONS.md— the flywheel and corrections documented, with the two release-time gates named.