8000 Evaluate Jev re-ranking for brain vector retrieval · Issue #47 · simon-inkie/inkie-the-brain · GitHub
[go: up one dir, main page]

Skip to content

Evaluate Jev re-ranking for brain vector retrieval #47

Open

Description

@simon-inkie

Original Linear description

Outcome

Use Jev to re-rank a bounded shortlist from brain vector retrieval, so coordinator context contains only relevant, provenance-preserving excerpts rather than a broad unfiltered result set.

Flow

  1. Build a compact search input from the latest directive plus bounded current-task context.
  2. Run existing vector retrieval for a small shortlist.
  3. Ask Jev to select a candidate ID or no_match from code-supplied IDs, titles and excerpts.
  4. Inject at most the selected one or two excerpts, with source path and timestamp.
  5. If probability/confidence is below the evaluated threshold or returns no_match, make a deterministic second retrieval pass: broader K and the latest directive alone. Re-rank the union.
  6. If still unsuitable, inject no brain result and continue normally.

Guardrails

  • Jev ranks supplied candidates only. It cannot create a query, source or fact.
  • Preserve candidate IDs, paths and timestamps in every result.
  • Cache by message ID, index version and candidate hashes.
  • Use only bounded task context, never raw full conversation history.
  • A low-confidence result is a retrieval abstention, not a claim that no relevant brain content exists.

Acceptance

  • Offline evaluation against labelled historical recall tasks compares baseline vector ranking with Jev re-ranking.
  • Report candidate recall, top-1 relevance, false-positive context injection, latency, input tokens and cost.
  • Thresholds are calibrated on those results, not copied from a generic example.
  • Feature is disabled by default until it meets its measured quality and cost gate.

Design note: /home/simon/brain/ideas/2026-09-22-jev-brain-retrieval-reranker.md

Origin

Migrated from Linear: INK-931. The original wording is retained except the disclosed privacy and rendering edits below.

Current scope and qualifications

The description above is the original Linear account, including its historical findings and scope. It is not a fresh claim that every described defect still exists.
Current Linear state is Backlog. Prior source review recorded open, with this remaining scope:

  • Run approved held-out recall evaluation and derive threshold before enabling; reuse in INK-963.

Publication transfers the task; it does not authorise live recovery, deployment, spending or configuration changes.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestmajor-featureA substantial feature or new capability

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0