Repository navigation
systemone: vote on the payload, never the instruction prose (#1000, #1001) - #1007
Merged
Merged
Conversation
…rose Two defects in the System One vote, both filed with their own measurements (#1000, #1001): #1000 — a question's `instructions` were joined into the retrieval query, so writing the instruction well DESTROYED accuracy: 0.9867 empty vs 0.9733 short vs 0.6700 criteria-rich on the same 300 held-out SMS messages (4,000 train, shuffle seed 7). A 32-point swing from wording alone. #1001 — an object `state` contributed only its `query` field, so `{"message": t}` voted on the instruction prose at the spam base rate: 0.1733 accuracy against 0.9667 for the same message as a string or as `{"query": t}`. The vote text is now exactly what the question points at: - every `path` reference the instructions name in backticks, resolved against the instructions' own string fields first (the data-field pattern xerj-rerank's stage and the provider docs use) and then against the state by dotted path (the jev-reranker shape). The state does not ride along when references resolve; - a reference that resolves to STRUCTURE, not text, is skipped: the rubric object jev-reranker >= 0.1.2 ships with its relevance preset is judge rules, not retrieval vocabulary. Embedding its string leaves is exactly the #1000 defect; erroring would break a documented client entry point (relevance_rerank), which 422'd before this rule ("`rubric` resolves to nothing") — proven live against 0.1.2 before the fix; - a question with no references votes on the state alone: a string whole, an object by ALL its string leaves, whatever the keys are named; - references that resolve to nowhere, and a payload with no text at all, are 422s naming the question — never a confident vote on text the question never saw. Also fixed here: `no_support()` interpolated `k` and the question ids into each other's format slots ("for question10 (k=`q1`)"). Measured after the fix (fresh-loaded history, same 300 held-out): - all four instruction wordings: 0.9767 identical - all four state shapes (string / {"query"} / {"message"} / {"msg","id"}): 0.9767 identical - both 422 shapes fire naming the question and path - acceptance gate, unmodified pip jev-reranker 0.1.2, both client entry points: neutral-query gap 0.8542 (spam 0.9447 / ham 0.0905), preset path identical to the constructor path (the rubric contributes nothing); deliberately spammy query gap 0.6087 — the query joins by client design since 0.1.2 references `query` in backticks, and the vote still separates The 0.9767 sits ~1 point under the issues' 0.9867 because the history was re-indexed fresh; the invariant the issues demanded is the identity across arms, which is exact. Tests: six new fail-before tests in systemone_http.rs (the five wording/ shape/422 tests failed `[0.0, 0.0, 0.4545, 0.0]` on the old code; the rubric test 422'd). Gates: xerj-api 272 lib + suites green, clippy 0, fmt clean, ES-YAML 1376 passed / 0 failed / 3 skipped (ci-test profile, private port, throwaway dir). Docs: DECISIONS.md contract rewritten; gate README re-dated with the 0.1.2 two-entry-point run; gate_run.py now exercises both entry points and the spammy query, its stale query-neutrality comment replaced with the true one; blog loss box rewritten — "inherent to a retrieval vote" was wrong, it was these two defects. Reference-code mandate: skipped — the change is confined to this repository's own code (systemone_api.rs) and its semantics are specified by the issues' own measurements, not by an external design. Fixes #1000 Fixes #1001
Cutting v1.0.0-rc.76 (b9d75dd) without rolling ROADMAP.md forward tripped two docs_capability_lists guards — exactly the drift issue #298 says cannot happen: - the_roadmap_review_line_names_the_latest_cut_release: line 5 still said the statuses were verified against v1.0.0-rc.75; - the_roadmap_next_release_has_not_already_been_cut: the next-release heading still advertised rc.76, and its narrative still described rc.75 as the last cut with an "In flight for rc.73" list two releases stale. Re-review: - review line -> 2026-09-21 against v1.0.0-rc.76 and main; - next-release heading -> rc.77, with an honest rc.76 summary from its CHANGELOG entry (System One wire + /_decide, WAND 2.0x, xerj code / corpus commands, FiQA datasets, .deb assets, the rerank 0.3822 fix); the s 8000 tale rc.72 narrative and rc.73 in-flight list are dropped — CHANGELOG is the history, the roadmap is the state; - in-flight for rc.77: the #1000/#1001 vote-text fix (this PR), #965 segment object storage, #950 RSS; - "Shipping today" gains the System One wire (shipped in rc.76, gate- verified): the vote, the benchmark numbers, the not-a-judge-model caveat. cargo test -p xerj-engine --test docs_capability_lists: 15 passed, 0 failed (2 failed before this commit).
… and xc_ matched nothing Main's "Reference-coding toolkit" job has been failing since bb6418b (#977) with "unexpected argument 'xc_' found": the step passed four positional test-name filters to one cargo test invocation, and cargo test accepts exactly one. The #977 branch run that would have caught it was cancelled before reaching this step, so it landed red on main. Two defects in the original line, found while verifying the fix: 1. cargo test takes ONE filter. The four filters are now a for-loop of separate invocations. 2. The filter "xc_" matched ZERO tests from the day it was written. The ported autoindex tests live at xc::tests::… — the substring "xc_" appears in no test name. The filter is xc:: now, and the 16 tests it was meant to run finally run in CI. Verified locally, the exact loop CI runs (--profile ci-test, same four crates): xccode 49 passed xc:: 16 passed published_schema 1 passed help_text 2 passed total 68 passed, 0 failed
…y the job never builds Second defect bb6418b left in the Reference-coding toolkit job, found by the first run that got past the cargo-filter fix: the "Published MCP schema drift" step it ADDED runs scripts/mcp-schema-check.sh, whose default XERJ_BIN is engine/target/release/xerj — and this job builds only with --profile ci-test, so the step failed "MCP server binary not found" on every run it ever reached (which, until the previous commit, was never: the cargo parse error killed the job first). The interaction is already solved this way in the smoke job (ci.yml, "XERJ_BIN must be set explicitly: this job builds with --profile ci-test (#331)"): point XERJ_BIN at target/ci-test/xerj, which the Toolkit-tests loop itself compiles (cargo test builds the bin target). Verified locally against the loop-built ci-test binary: served: 11 tools ... RESULT: PASS — published schema matches the binary (11 tools)
This was referenced Sep 21, 2026
xerj-org
added a commit
that referenced
this pull request
Sep 21, 2026
…low the in-cut version stamp PR #1016 is the first release/* PR since the gates landed, and three of them failed — each a real catch, not noise: 1. docs_capability_lists::the_roadmap_next_release_has_not_already_been_cut and ::the_roadmap_review_line_names_the_latest_cut_release: the cut commit (55bffba) stamped the CHANGELOG but did not roll ROADMAP.md forward — line 5 still reviewed against rc.76 and the "Next release" heading still advertised rc.77. rc.76 dodged this only because it was cut directly on main (b9d75dd), which never runs the PR gates. ROADMAP.md now: review line names rc.77 (with the scope of this review stated — the Next-release section and shortlist were re-reviewed; the Shipping-today claims were last live-verified against rc.76), the heading points at rc.78, the in-flight list is replaced (the old one listed eight issues that are all closed now, and characterised #950 as "RSS feeds" — it is resident-set memory), and the "**Open defects" shortlist the release-notes gate parses is back, carrying only the five issues that are open today (verified via the API): #1015, #950, and the trackers #941, #874, #298. 2. release-notes-gate.sh check 1a ("found no linked issues in ROADMAP.md's '**Open defects' shortlist"): the shortlist was dropped from ROADMAP.md in 5f3b6ea (rc.72-era rewrite) while the gate that requires it shipped in rc.71 (#870). Restored by (1) above; verified locally that every cited issue is open and that PR #1007 is covered via the issue it closes. 3. landing-constants-guard.sh ("version stamps disagree with the newest released tag"): the check required footer stamps to equal the newest tag, which a release PR can never satisfy — the footers are stamped with the version being cut before that version's tag exists. rc.76 dodged it the same way as (1). The allowed set is now {newest released tag, engine/Cargo.toml version}: a footer matching the version of the tree it ships in is self-consistent, not drift, and the set collapses to just the tag on every non-release branch. Applied to both the footer and the llms*.txt "Current release" checks. Verified locally: landing-constants-guard.sh all checks pass; release-notes- gate.sh PASS (freshness, shortlist states, coverage incl. #1007-via-#1000); docs_capability_lists 15/15.
neoneye
pushed a commit
to agent-memory-atlas-archive/xerj-org--xerj
that referenced
this pull request
Sep 22, 2026
The stateless-index and reader-fairness release. Two headline changes, each with a fail-before reproduction; everything else is a fix: - storage.backend = "s3" starts for real (xerj-org#965, PR xerj-org#1008): the whole segment family packs into ONE immutable ZBM1 bundle object (footer = the ZCM1 manifest + absolute offsets; trailer adapted from quickwit's split-footer bundle, Apache-2.0, approach only). One object + the catalogue PUT is ~18% of R2's free Class-A tier at a 30 s flush interval where per-file PUTs would be ~899%. snapshot.json is the publication point; merges publish before retiring; a FRESH node on an empty data dir boots from the bucket and serves full-text over hydrated families (journey test). Fail-before: the round-trip test panicked "the flush must publish the snapshot catalogue". Honest limits stay in docs/OBJECT_STORAGE.md: single-writer v1, WAL local, in-memory packs, simulation-pinned crash consistency. Config 127 -> 128 - readers no longer starve under sustained ingest (xerj-org#1013, PR xerj-org#1014): the seqlock token was validated at the END of a search, demanding a writer-free interval spanning the whole scan — under flush pressure those stop existing and search/delete_by_query 500'd at their deadline (reproduced twice in the xerj-org#950 profiling run). Validation now happens at the CAPTURE boundary with the evenness check; readers need a writer-free instant, not a writer-free search. Fail-before: churn test failed at its 2 s deadline with the production error; now passes in 0.37 s with exact totals (723/723 lib tests, ES-YAML 1376/0) - /v1/systemone votes on the payload, never the instruction prose: 0.9867 empty vs 0.6700 criteria-rich instructions was a 32-point swing from wording alone (xerj-org#1000); {"message": ...} voted at the spam base rate, 0.1733 vs 0.9667 as a string (xerj-org#1001). All eight wording/shape arms now score 0.9767 identically (PR xerj-org#1007) - the console tour's pill assertions raced the route hand-off (xerj-org#1011) — aria-busy never goes 'true', so the wait matched the PREVIOUS route's DOM; failed only on slow runners (bit PR xerj-org#1008's CI) - CI honesty: the reference-coding toolkit's tests never ran (one filter per invocation; xc_ matched nothing — green on zero tests), and its MCP schema gate looked for a release binary the job never builds; the roadmap was re-reviewed against rc.76 (release-drift guard) - follow-up filed: xerj-org#1015 (flush guard spans the whole finalize — a latency consideration now, not a starvation hazard) Version stamps: engine/Cargo.toml + Cargo.lock at 1.0.0-rc.77; landing footers V1.0.0-RC.77 (51 files). Two historical rc.76 mentions kept on purpose: the CHANGELOG section and the "since rc.76" feature-landing reference in landing/answers/rebuild-reference-corpus-index.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
/v1/systemone's vote text was instruction prose plus (for object states) thequeryfield. Now a question votes on exactly what it points at: its backtick references (instructions data fields first, then state paths), or — with no references — the state's own string content. A reference that resolves to structure rather than text (the rubric objectjev-reranker≥ 0.1.2 ships with its relevance preset) is acknowledged and skipped. References resolving to nowhere, and textless payloads, are 422s naming the question. Also fixesno_support()'s swapped format args.Why — the issues' own numbers
statesilently drops every field exceptquery, answering from the instruction alone #1001: object state contributed onlyquery→{"message": t}= 0.1733 (spam base rate) vs 0.9667 as a string.After (re-measured on this branch, fresh-loaded history)
{"query"}/{"message"}/{"msg","id"}q1+typo.pathq1(The absolute number sits ~1 point under the issues' 0.9867 because the history was re-indexed fresh; the invariant the issues demanded is the identity across arms, which is exact.)
Acceptance gate — unmodified pip
jev-reranker0.1.2, both client entry points:relevance_rerank()preset (rubric object in state): identical to the constructor path — the rubric contributes nothing, which is the /v1/systemone: the question's instructions are concatenated into the retrieval query, so a well-written instruction destroys accuracy #1000 lesson enforced`query`in backticks, so the query is named payload by client design; the vote still separatesThe rubric rule (found live, not invented)
The 0.1.2 preset's question says "Apply all remaining evaluation rules in
rubric" while shipping the rubric as an object in state. Pre-branch, that request 422'd ("rubricresolves to nothing") — a documented client entry point, broken. Embedding the rubric's string leaves would be exactly the #1000 defect (criteria-rich prose). So: structure references are acknowledged and skipped; only references resolving to nothing are errors. Fail-before for this is the live 422 against 0.1.2 (noted in the test's doc comment), nowa_structure_reference_like_the_clients_rubric_is_not_vote_text.Gates
systemone_http: 20/20 (six new fail-before tests; the wording test saw[0.0, 0.0, 0.4545, 0.0]on old code)audit.py,factcheck.py(0 findings on the article),claims_rules.pycleanDocs updated in-flight
docs/DECISIONS.mdcontract rewritten; gate README re-dated to the 0.1.2 run;gate_run.pynow exercises both entry points + the spammy query (its stale "query rides in state.query" comment replaced with the true client-design reason); the blog loss box no longer claims the query-pollution failure was "inherent to a retrieval vote" — it was these two defects, with the numbers.Reference-code mandate
Skipped, deliberately: the change is confined to this repository's own code (
systemone_api.rs), and its semantics are specified by the issues' own measurements, not by an external subsystem design.Fixes #1000
Fixes #1001