8000 feat(hub): agents rate hub corpora (xerj_corpus_feedback, xerj hub feedback, automatic reviews) by xerj-org · Pull Request #1886 · xerj-org/xerj · GitHub
[go: up one dir, main page]
Skip to content

feat(hub): agents rate hub corpora (xerj_corpus_feedback, xerj hub feedback, automatic reviews) - #1886

Merged
xerj-org merged 4 commits into
mainfrom
client/corpus-feedback
Oct 20, 2026
Merged

feat(hub): agents rate hub corpora (xerj_corpus_feedback, xerj hub feedback, automatic reviews)#1886
xerj-org merged 4 commits into
mainfrom
client/corpus-feedback

Conversation

@xerj-org
@xerj-org xerj-org commented Oct 19, 2026
Copy link
Copy Markdown
Owner

This is the client half of the Agent score on hub.xerj.org: AI agents rate the hub corpora they use, kept separate from human hub ★. The endpoint lands on corpus-hub in #1881 (POST https://hub.xerj.org/api/corpus-feedback).

What sends a review

when basis what it carries
xerj hub pull <slug> auto, once per corpus per week whether the pull worked and whether it was slow, plus the pin. No score. It also prints the line asking the agent to rate the corpus once it has used it.
MCP xerj_hub_ask / xerj_hub_def answer auto, once per corpus per week, in the background found and usefulness from whether the answer had hits (never 5, which only the agent can give), freshness from the pack's age, the question's shape only
MCP xerj_corpus_feedback (new tool) agent usefulness 1-5, found, freshness, agent family/version, pack, query template, coverage gaps, problems + note. Sends by default; confirm:false previews
xerj hub feedback <slug> --usefulness N --found yes|partial|no [...] agent the same, from a shell (--dry-run prints the JSON)

There's no prompt, like field reports. The server keeps one review per install per corpus per week. An agent review replaces the automatic one, and an automatic review never overwrites an agent's.

Privacy

  • The raw query never leaves the machine. normalise_template keeps placeholders and a fixed list of structure words, and every other word becomes <term>, <num>, <str> or <ref>. The server repeats the same rule.
  • The install id is random, made locally (<hub home>/corpus-feedback.json), and never tied to a person.
  • The agent family comes from the MCP client's clientInfo.name or tool env markers (CLAUDECODE, CURSOR_AGENT, …), and XERJ_AGENT_FAMILY overrides it.

Opt-out and safety

  • --disable-feedback / XERJ_DISABLE_FEEDBACK=true turns all of it off.
  • Automatic sends skip a process that verifies packs under a non-production key (XERJ_HUB_PUBKEY, which the tests set) unless XERJ_CORPUS_FEEDBACK_URL is set. So the test suite never posts.
  • An automatic send times out after 4 s and never fails the command.

Docs

  • AGENTS.md has a new short section, "If you used a hub corpus, rate it".
  • landing/llms.txt has the same line under Field report, and the tool count goes from seventeen to eighteen.
  • MCP instructions are updated.
  • mcp-tools.json was regenerated with scripts/mcp-schema-check.sh --write (18 tools).

Tests (local, quick profile)

  • xerj-common corpus_feedback: 6 Rust payload tests (field names, caps, refusals, template never carries the raw query, signals → scores, family detection, ISO week).
  • xerj-hub feedback: state/weekly claim, CLI arg parsing, answer → auto review, auto-send gating, URL refusal. The full xerj-hub suite passes (275 unit + integration).
  • xerj-mcp passes 93 unit tests (tool count 18/15), plus published_schema_drift and hub_tools.
  • cargo clippy -p xerj-common -p xerj-hub -p xerj-mcp --all-targets -D warnings is clean, and scripts/seo/test_llms_field_report.py passes.
  • End to end: the MCP xerj_corpus_feedback call against a local wrangler pages dev of hub: /api/corpus-feedback — structured agent reviews per corpus (Agent score backend) #1881 was stored and aggregated.

@xerj-org
feat(hub): agents rate hub corpora — xerj_corpus_feedback, xerj hub f…
8aafbb0 
…eedback, automatic reviews after pull and MCP hub answers

Adds the client half of hub.xerj.org's Agent score (POST
https://hub.xerj.org/api/corpus-feedback, corpus-hub #1881).

- xerj-common::corpus_feedback: the review payload with the server's caps;
  the query is reduced to its shape (placeholders + structure words, every
  other word <term>), never sent raw; random local install id; ISO week.
- xerj hub feedback <slug> --usefulness 1-5 --found yes|partial|no [...]
- xerj hub pull: one automatic pull note per corpus per week (worked? slow?,
  no score) and the line asking the agent to rate the corpus once used.
- MCP: new tool xerj_corpus_feedback (sends by default, confirm:false
  previews); xerj_hub_ask / xerj_hub_def send one automatic review per
  corpus per week in the background (hits or not, pack age; never 5/5).
  The agent's own review replaces the automatic one on the server.
- All of it honours --disable-feedback / XERJ_DISABLE_FEEDBACK; automatic
  sends skip a non-production hub key (tests) unless
  XERJ_CORPUS_FEEDBACK_URL is set; never block more than 4s or fail the
  command.
- AGENTS.md and landing/llms.txt tell agents to send it; mcp-tools.json
  regenerated (18 tools).
@cla-bot cla-bot Bot added the cla-signed label Oct 19, 2026
@xerj-org
Merge branch 'main' into client/corpus-feedback
af0bc84
@xerj-org
egress inventory: classify xerj-hub/src/feedback.rs and publish the c…
c7da42c 
…orpus-feedback path

feedback.rs POSTs structured corpus reviews to hub.xerj.org/api/corpus-feedback,
which tripped every_outbound_client_in_the_engine_is_on_the_published_list.
Classify it as a client path (`xerj hub feedback`) and add its row to
docs/RERANK.md "Every way data leaves a XERJ node" and to the air-gapped recipe,
naming --disable-feedback / XERJ_DISABLE_FEEDBACK.
@xerj-org
style: rustfmt the feedback.rs egress inventory row
a5dd759
This was referenced Oct 20, 2026
Merged
Merged
Merged
@xerj-org
xerj-org merged commit becab48 into main Oct 20, 2026
43 of 46 checks passed
@xerj-org
xerj-org commented Oct 20, 2026
Copy link
Copy Markdown
Owner Author

Xerj Merger made the fix: reviewed the full diff. This is the client half of the hub Agent score: xerj hub feedback, the xerj_corpus_feedback MCP tool, and automatic reviews after xerj hub pull and the MCP hub answers, sent at most once per corpus per week with a 4 s cap. The only egress is one JSON POST to hub.xerj.org/api/corpus-feedback (or XERJ_CORPUS_FEEDBACK_URL). It carries no query text: normalise_template turns every non-structure word into <term>. The egress inventory, RERANK.md and the air-gapped recipe all list it. Worth an explicit owner call: the automatic reviews are on by default (opt out with --disable-feedback / XERJ_DISABLE_FEEDBACK), and AGENTS.md / llms.txt tell agents to rate corpora without asking the user. Verified locally at a5dd759, merged with main 5585819 (in a combined tree with #1861, #1910, #1700 and #1746): fmt, clippy -D warnings and cargo test --profile ci-test --workspace are all green. That run was ahead of the CI backlog (the hosted Build + Test was cancelled), on the owner's instruction.

@xerj-org Cursor
xerj-org commented Oct 20, 2026
Copy link
Copy Markdown
Owner Author

Xerj Merger made the fix: the client now sends structured corpus feedback automatically after hub pull and MCP hub queries, adds the xerj_corpus_feedback MCP tool and xerj hub feedback, and respects --disable-feedback / XERJ_DISABLE_FEEDBACK.

@xerj-org xerj-org mentioned this pull request Oct 20, 2026
Merged
2 of 8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

cla-signed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@xerj-org
0