Repository navigation
bench(ask-plan): latency arm — #1056's p50 gate measured, 1.129 ms (PASS) - #1079
Merged
Merged
Conversation
The one #1056 gate line still unmeasured after #1076 merged: 'p50 <= 300 ms CPU budget' per POST /_ask. The ask arm times every call but scores F1 only — the milliseconds were never recorded. New scripts/latency_arm.py (python3 stdlib, same conventions as ask_arm): one warmup pass + N measured passes over the same 230 pairs.jsonl prompts on the same booted node, per-request wall-clock ms to latency-raw.jsonl, nearest-rank p50/p90/p95/p99/max + a gate verdict to latency-summary.json. Honest scope stated in the file and the summary: wall-clock over loopback on an idle node is an UPPER BOUND on CPU time, not CPU time; prompts carry their gold index so catalog routing is out of the path (it has its own acceptance line); warmup reported separately so the steady-state choice is visible, not flattering. Measured run (release build of main f0b96d6, private :9681, throwaway data dir, fixtures loaded by the unmodified ask_arm.py loader, selfcheck green, run committed verbatim in results/2026-09-30-latency-gate/): measured n=690 (230 x 3 passes, all HTTP 200) p50 1.129 ms · p90 1.883 · p95 2.033 · p99 2.852 · max 155.719 warmup p50 1.102 ms · max 468.927 (first-touch) gate p50 <= 300 ms: PASS — 265x under the line Caveats kept in the README, not buried: the fixture indices are small (655 / 3,000 / 1,704 docs) — the p50 is a fact about THIS fixture, not a promise about larger corpora; the gate row claims wall-clock, nothing about CPU. README: the DESIGN banner is two runs stale — gate 2 (F1 0.9975, 0 invalid DSL, 0 nondeterminism, recorded in PR #1076's body from an unmodified scratch copy whose raw files were never committed — said plainly in the row) and now gate 4 each carry MEASURED rows; the NOT-measured list shrinks to what is actually unmeasured (token arms, catalog routing). run.sh gains step 5 so a full run captures latency alongside F1 from here on.
This was referenced Oct 1, 2026
neoneye
pushed a commit
to agent-memory-atlas-archive/xerj-org--xerj
that referenced
this pull request
Oct 1, 2026
The packaging half of the ten-item program's rule (discussion xerj-org#1054): a benchmarks/<name> directory with raw results plus a blog post with the losses left in. The gate directories landed through xerj-org#1079, xerj-org#1085, xerj-org#1086, xerj-org#1087, xerj-org#1088 and xerj-org#1090; this is the post. Structure follows the Jev posts (verdict first, wins then losses with readings, then what the next window does about each loss). Every number is verbatim from the committed gate artifacts: - PASS: /_ask macro-F1 0.9975 over 230 pairs, 0 invalid DSL, 3-run determinism, wall-clock p50 1.129 ms vs the 300 ms bar (xerj-org#1076, xerj-org#1079); xerj_map unknown-field 400s 20/30 guessed -> 0/30 map-informed, 100% vs the >=80% bar (xerj-org#1074, xerj-org#1086); calibration FiQA pair-level ECE 0.3109 raw -> 0.0088 held-out vs the <=0.10 bar, temperature 0.3163 published as the failing comparison arm, $0.2125 of provider calls over 648 queries (xerj-org#1080, xerj-org#1087). - FAIL, published as such: the lexical judge loses to its own first stage everywhere measured (0.7045->0.5967, 0.3419->0.2924, 0.2382->0.1650 vs the 0.30 FiQA bar) and stays opt-in with no quality claim per its own rule (xerj-org#1077, xerj-org#1090); the flywheel freezes as shipped (12 of 2,000 cached, xerj-org#1094) and forces 19.72% coverage at 0.9953 accuracy vs the 80%/0.97 bars (xerj-org#1075, xerj-org#1088); the tier-2 decide head misses SMS 0.9193/0.287 vs 0.95/0.05 and AG News 0.2599 vs 0.85, latency 8.4 ms PASS, ships behind the flag with the numbers published (xerj-org#1072/xerj-org#1073, xerj-org#1088). - SPLIT: stemming +0.0160/+0.0179 BM25 nDCG@10 PASS, zero-hit 25->15 vs <=10 FAIL — the 15 have no stem anywhere in the corpus (xerj-org#1070, xerj-org#1085). - The gate that caught the stock release binary built without the decide-local cargo feature (flag a no-op, stock-binary-noop.txt) is told with the fix queued ahead of the cut, not claimed as done. - Honest gaps stated, not smoothed: the /_ask token arm designed not run, the _watcher detection gate not yet measured, no HF URL for xerj-decide-v1. Four issues filed from the measurements: xerj-org#1091-xerj-org#1094. Mechanics: hand-written page at landing/blog/the-rc80-gates.html, registered in scripts/seo/pagedata.py (published/updated 2026-09-30), card prepended to the blog hub (hub dateModified bumped), head and footer owned by fix_heads/fix_links, og falls back to the site card.
neoneye
pushed a commit
to agent-memory-atlas-archive/xerj-org--xerj
that referenced
this pull request
Oct 1, 2026
The rc.80 release PR, docs-only on top of main (all engine code rides the merged window PRs; nothing here touches engine/crates or xerj-ux). rc.79 was folded into this cut by direction; no rc.79 tag exists. - CHANGELOG: [Unreleased] cut over to ## [1.0.0-rc.80] - 2026-09-30 with the measured-gates theme (three gates passed outright, three failed and ship opt-in/behind a flag with numbers published, one split); xerj-org#1095's own entry moves from [Unreleased] into the section unchanged; adds the citations the coverage gate was missing (xerj-org#1078 tool inventory, xerj-org#1079 ask-plan latency arm 1.129 ms, xerj-org#1083 egress-inventory classification, xerj-org#1089 console UX review) plus xerj-org#1097 (merged with no entry of its own). - ROADMAP: review line bumped to 2026-09-30 against rc.80; the ten-item program (xerj-org#1054, xerj-org#1055-xerj-org#1064) recorded as executed with every verdict; the semantic-detections row rewritten (the 'no alerting' text was false since xerj-org#1082); Next release -> v1.0.0 GA with the rc.80 cut record and a live-verified five-defect shortlist (xerj-org#1091-xerj-org#1094, xerj-org#1100) — xerj-org#1098/xerj-org#1099 were fixed and closed by xerj-org#1095 inside this release. - landing: all 53 V1.0.0-RC.78 stamps -> RC.80 (landing-constants-guard matches the tree version); data-latest-tag rc.78 -> rc.80. - llms.txt + llms-full.txt: the watcher caveat rewritten to the rc.80 reality (one real watch shape, everything else 501) with the earlier-version note kept in the open. - content/answers/does-xerj-support-s3-alerting-plugins.md: the alerting half rewritten (PARTIAL since xerj-org#1082) and the packs half corrected (record packs real since rc.78); updated bumped to 2026-09-30. - release-notes-gate.sh: an empty open-defects shortlist is a real state (tracker emptied 2026-09-29) — distinguish 'section moved' from 'section lists none'; verified against the live tracker. - engine/Cargo.toml 1.0.0-rc.78 -> 1.0.0-rc.80; Cargo.lock version strings only (no dependency moves — xerj-org#1095/xerj-org#1097 added none). docs_capability_lists: 15 passed, 0 failed. landing-constants-guard: all checks passed. release-notes-gate: PASS, all window PRs cited.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the last unmeasured #1056 gate line. Packaging-rule follow-up to #1076 (plan #1054, item #1056).
What shipped
scripts/latency_arm.py— the latency arm the harness lacked: the ask arm times every call but scores F1 only; the milliseconds were never recorded. One warmup pass + N measured passes over the same 230pairs.jsonlprompts on the same booted node; per-request wall-clock ms raw, nearest-rank quantiles + gate verdict summarized. Stdlib-only, same conventions asask_arm.py(honest stop on endpoint-absence, nothing fabricated).run.shstep 5 — a full run now captures latency alongside F1.results/2026-09-30-latency-gate/— the measured run, committed verbatim.Measured (release build of main
f0b96d634, private :9681, throwaway data dir, unmodifiedask_arm.pyloader, fixture selfcheck green):Gate: p50 ≤ 300 ms — PASS, 265× under the line.
Scope honesty (in the README, not buried): wall-clock over loopback on an idle node is an upper bound on the gate's CPU budget, not CPU time; prompts carry their gold index so catalog routing is out of the path (own acceptance line); the fixture indices are small (655 / 3,000 / 1,704 docs) — the p50 is a fact about this fixture, not a promise about larger corpora.
README banner fix while here: it still said DESIGN — NOTHING MEASURED, two runs stale. Gate 2 now carries its MEASURED row too (F1 0.9975, 0 invalid DSL, 0 nondeterminism over 230×3 — recorded in #1076's body from an unmodified scratch copy whose raw files were never committed; the row says so). The NOT-measured list shrinks to what is actually unmeasured: the token arms (gate 3) and catalog routing.