Repository navigation
hub backlog: agent-session-trajectories (planned, demand from #1138) - #1210
Merged
Merged
Conversation
Chunxiao Wang (Assay / nautilus-compass) offered two retrieval/eval recipes in #1138 and named the corpus that would change their eval answers: real multi-session agent trajectories with failure turns kept verbatim. Zero of the 99 live manifests and none of the other backlog rows are session or trajectory data, so the want gets a named slot. Status planned, lane B, pinned refresh; source hint points at their repo (eval artifacts today, not yet a trajectory dump) and the donation path stays open via CONTRIBUTING.md. validate_hub.py: backlog 132 rows, 99 live, OK.
chunxiaoxx
added a commit
to chunxiaoxx/xerj
that referenced
this pull request
Oct 8, 2026
…gent trajectories, failure turns intact) Domain: an agent operating a production multi-agent engineering org queries this corpus for precedent on failed approaches, their diagnosis, and the fix that worked. Records are unique by id (nst-<session-key>); 70 envelopes -> 70 records, no alias resolution (single source). Topics: eval-judging 46 / infra-diagnosis 13 / ledger-audit 6 / ops-misc 5. Provenance: our own production ops records (CC-BY-4.0) — no customer data, no third-party content; sanitized via a 5-step pipeline + 10% human sample audit (7/7 pass, seeded); final scan clean (0 credential/PII/schema failures, tools/xerj_pack_scan.py selftest 5/5). Baseline attribution: contains our evaluation-process records against SWE-bench-family benchmarks — our own process and readouts only, no benchmark data redistribution. Demand anchor: issue xerj-org#1138 + hub backlog slot (PR xerj-org#1210). Eval-answers offer stands: first seed triggers an independent assay eval (preregistered criteria, three-state answers, UNVERIFIABLE wall) with the full answer set published.
thomas-villani
pushed a commit
to thomas-villani/xerj
that referenced
this pull request
Oct 9, 2026
Supersedes the .contributors hunk of PR xerj-org#1257, which carried the signature on a stale base and against the corpus-hub branch, where the CLA bot never reads it (it resolves .contributors from main). chunxiaoxx signed in PR xerj-org#1257 itself; the signature enables the agent-session-trajectories record pack (PR xerj-org#1255, wishlist slot from xerj-org#1210, demand anchored to xerj-org#1138) and the pending guest-post branch.
thomas-villani
pushed a commit
to thomas-villani/xerj
that referenced
this pull request
Oct 9, 2026
…+ 1cfad62) The first external-contributor pack (chunxiaoxx / Nautilus, PR xerj-org#1255, CLA signed via xerj-org#1263, demand anchored to xerj-org#1138, wishlist slot xerj-org#1210): 70 sanitized multi-session agent trajectories, failure turns kept verbatim, CC-BY-4.0, single-source identity (70 envelopes -> 70 records, measured). Main needs the identical copy because scheduled workflows fire from the default branch only — the daily pack-publish cron builds and signs the pack from tools/packs/agent-session-trajectories/recipe.toml with the PACK_SIGNING_SEED_AST secret, verifying against the committed .pub before each publish. corpus-hub carries the same files for hub-side validation (PR xerj-org#1264). No engine change: recipe + stats + key + one workflow job.
neoneye
pushed a commit
to agent-memory-atlas-archive/xerj-org--xerj
that referenced
this pull request
Oct 10, 2026
Chunxiao Wang (Assay / nautilus-compass) answered our practitioner invitation in xerj-org#1118 and xerj-org#1138 with six production patterns: topic slicing beating token windows on recall@1 by double digits, an embedder bakeoff where the MTEB pick lost recall@1 (+0.46pt r@5 vs +2.75pt r@1 zh), qid grouping against eval leakage, three-valued retrieval (Brier 0.069 with abstention), cached-input variance attribution (0.0pp judge-side), a 0.87 classifier nearly matched by a 0.78 always-yes baseline at n=60, and the 4x unpaired overstatement. Every measurement in the post is theirs and attributed. The XERJ-side section answers the question they asked back: where the registry is thin. Checked, not guessed: 0 of 99 live manifests and 0 of 132 backlog rows are session or trajectory data; our chunk unit is file/section, not the semantic topic their recall@1 gain needs; the memory features store curated facts with BM25 recall and no tried-and-failed marker. The want has a named backlog slot (PR xerj-org#1210 on corpus-hub) and the donation path is open. New page + blog index card + pagedata entries + sitemap (184 URLs). seo_lint: 0 gate violations.
neoneye
pushed a commit
to agent-memory-atlas-archive/xerj-org--xerj
that referenced
this pull request
Oct 13, 2026
…eeded Issue xerj-org#1138 is a community contribution thread (Chunxiao Wang / Assay): two eval/retrieval recipes plus a corpus wishlist, asking in return only for an honest note on where our index is thin against session-memory workloads. All of its original asks already landed: the honest-gap comment, the recipes/blog page (PR xerj-org#1211, this file), the hub backlog slot (PR xerj-org#1210), and the donated corpus pack itself (PR xerj-org#1255, release pack-agent-session-trajectories-2026-10-09, wiring xerj-org#1264). Root cause of this change: the published blog page was honest on 2026-10-07 but has since gone stale. §04 states, flatly, that the planned session-trajectory corpus "exists in any form. It does not" — true when measured, false now that PR xerj-org#1255 merged (2026-10-08) and the signed pack was released (2026-10-09). A page carrying a dateModified must not keep asserting zero when the number is no longer zero. What changed: - Added a dated UPDATE note (2026-10-11) under §04 recording the landed donation: 70 sanitized multi-session trajectories with failure turns intact, CC-BY-4.0, via the documented contribution path, and the maintainer-published signed record pack. Links to xerj-org#1255, the release tag, and xerj-org#1264. - Softened the "Not claimed … it does not" sentence to "as of the 2026-10-07 registry check … It did not then", pointing to the update. The §04 counts themselves are left verbatim — the honest record is that the number was zero when we checked. - Noted that the one still-open item, Chunxiao's independent eval against the seeded corpus, has not yet been published, and claimed nothing about its result. - Bumped dateModified / pagedata updated to 2026-10-11. Before: page asserts the corpus does not exist (stale since 2026-10-08). After: page records the seeding, keeps the original measurement intact, and still claims nothing unverified. No engine code touched; ES-YAML gate unaffected. SEO gate green locally (fix_heads/fix_links/seo_lint/build_articles/gen_sitemap/mk_og_card/ test_article_schema all pass; 0 gate violations). Advances xerj-org#1138 (partial). Remaining and out of our hands: Chunxiao's offer to publish an independent recompute (pre-registered criteria, three-state verdicts, UNVERIFIABLE wall) against the now-seeded corpus stands on their side and is not yet delivered; closing the issue is the maintainer's call once that eval lands or the thread is declared done.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a named backlog slot for the corpus #1138 asks for: real multi-session agent trajectories with timestamps and failure turns kept verbatim.
Demand anchor: Chunxiao Wang (Assay / nautilus-compass) said in #1138 they will switch their eval answers to such a corpus if it exists. It does not: zero of the 99 live manifests and none of the other backlog rows are session or trajectory data. The hub's chunking (file/section) is also the wrong shape for topic-sliced session memory, said plainly in the issue.