TraceRAG is an open-source, evidence-first NFL rules explainer. It resolves questions and play scenarios against a versioned rules corpus while exposing its retrieved evidence, citations, quality metrics, latency, and token usage.
The core keeps retrieval mechanics explicit and measurable. Framework integrations, including LangChain, remain replaceable adapters and are evaluated against the same benchmark.
- This README — project scope, setup, and architecture diagrams.
- CONTEXT.md — the precise meanings of ruling, evidence, citation, and abstention.
- Corpus guide, then Completing a Catch and Scoring Plays — concrete examples of what the system can know.
- Evaluation — scoring mechanisms, measured results, and their limits.
- Roadmap — completed work and planned improvements.
The browser experience frames each question as a play under review while keeping corpus scope and retrieval behavior visible.
A supported ruling shows the model's answer beside execution telemetry: retrieved passages, response time, token count, generation attempts, and the embedding and generation models. It then exposes the ranked passages and identifies which ones were accepted as citations.
TraceRAG currently provides:
- A typed FastAPI service with a health endpoint
- PostgreSQL 17 with the pgvector extension
- Reproducible Python dependencies through uv
- A smoke test and GitHub Actions workflow
- A versioned 2026 NFL rules corpus with authoritative rule references
- Deterministic Markdown ingestion and structural chunking
- Local
BAAI/bge-small-en-v1.5embeddings through FastEmbed - Exact cosine retrieval through PostgreSQL and pgvector
- A corpus status endpoint
- Grounded answer generation through Groq and
openai/gpt-oss-20b - Targeted recovery from intermittent structured-output validation failures
- Server-resolved citations and explicit abstention for unsupported rulings
- Retrieval, model, token-usage, and latency traces in answer responses
- A versioned evaluation set with retrieval, abstention, citation, latency, and usage metrics
- Case-specific answer-correctness rubrics and attributed review scores
- A responsive evidence-first web interface served by the API
- A milestone-based delivery roadmap
Retrieval remains available as an independent endpoint so its quality can be evaluated separately from answer generation.
Prerequisites:
- Docker with Compose
- uv
Install dependencies and run the tests:
uv sync
uv run pytestThe first embedding operation downloads an approximately 67 MB quantized ONNX model into the configured cache directory.
Start the complete local stack:
cp .env.example .env
# Set TRACERAG_GROQ_API_KEY in .env.
docker compose up --buildIn another terminal, index the corpus. The operation is idempotent and updates changed chunks while removing stale chunks from the active corpus season:
docker compose exec api tracerag-indexThen open:
- Web interface: http://localhost:8000/
- Health check: http://localhost:8000/health
- Corpus status: http://localhost:8000/corpus/status
- Interactive API documentation: http://localhost:8000/docs
Retrieve evidence without generating an answer:
curl -X POST http://localhost:8000/retrieval/search \
-H 'Content-Type: application/json' \
-d '{"question":"What makes a sideline catch complete?","top_k":3}'Generate a grounded ruling with citations and its execution trace:
curl -X POST http://localhost:8000/answers \
-H 'Content-Type: application/json' \
-d '{"question":"What makes a sideline catch complete?","top_k":3}'The generator receives passage identifiers and text, but not authority to create citation
metadata. TraceRAG resolves every cited identifier against the retrieved passages and abstains if
the model references evidence that was not retrieved. TRACERAG_GROQ_API_KEY is required only for
the answer endpoint; health, corpus, indexing, and retrieval operations remain local.
Generation defaults to the model's medium reasoning effort. It can be changed with
TRACERAG_GENERATION_REASONING_EFFORT, while
TRACERAG_GENERATION_STRUCTURED_OUTPUT_RETRIES controls the narrow retry budget for provider JSON
validation failures. Successful answer traces expose generation_attempts; exhausted provider
errors remain HTTP 502 responses rather than being converted into abstentions.
Run the deterministic retrieval benchmark against the indexed corpus:
docker compose exec api tracerag-evaluate --top-k 5The report includes hit rate, recall, reciprocal rank, and a result for every evaluation case. Unsupported questions are excluded from retrieval relevance metrics because they intentionally have no expected supporting document. Cases are tagged by scenario type, and the report includes the same aggregate metrics per tag.
Compare retrieval depth in one local run:
docker compose exec api tracerag-evaluate --top-k-sweep 1 3 5Answer generation is intentionally unavailable during a sweep so the command cannot accidentally multiply provider usage.
Answer evaluation is optional because it calls the configured generation provider once per case:
docker compose exec api tracerag-evaluate --top-k 5 --answersThat mode additionally measures abstention accuracy, citation-document precision, expected-document
coverage, latency, and token usage. Citation-document precision verifies that citations point to the
expected rule explanation. Expected-document coverage checks whether each answerable case answers
and cites exactly the expected document set. These automatic checks do not grade the ruling text.
The former grounded_ruling_accuracy JSON field is now expected_document_coverage; the per-case
field is now expected_documents_matched.
The v3.1 dataset includes correctness rubrics for all 40 answerable cases. Save a new answer report and export a review template:
docker compose exec -T api tracerag-evaluate --top-k 5 --answers > answer-report.json
uv run tracerag-review --report answer-report.json > answer-review.jsonRead each ruling, explanation, and retrieved evidence, then fill in the reviewer name,
review_kind (human or model_assisted), per-criterion booleans, and notes for failures. Score
the completed review offline:
uv run tracerag-review --report answer-report.json --review answer-review.json > correctness.jsonCorrectness requires passing every rubric requirement. Abstentions and provider failures on answerable cases score zero; incomplete reviews are rejected. Fresh model-assisted results and concrete failures are recorded in the published evaluation baseline, including its limitations. See the four separate answer-quality experiments for controlled prompt/model checks, the optional semantic gate, and full-set/holdout run status.
Stop the services with docker compose down. The PostgreSQL data remains in the named Docker
volume between restarts.
flowchart LR
ui["Evidence-first<br/>web interface"]
subgraph Ingestion["Corpus ingestion"]
docs["Versioned rules<br/>Markdown corpus"] --> parser["Loader +<br/>structural chunker"]
parser --> chunks["Rule passages<br/>with citations"]
chunks --> embed["FastEmbed<br/>bge-small-en-v1.5"]
embed --> store[("PostgreSQL<br/>+ pgvector")]
end
subgraph Query["Question answering"]
question["NFL rules question"] --> qembed["Query embedding"]
qembed --> retrieve["Exact cosine<br/>retrieval"]
retrieve --> evidence["Retrieved evidence<br/>and passage IDs"]
evidence --> generator["Groq generator<br/>openai/gpt-oss-20b"]
generator --> validate{"Citations reference<br/>retrieved evidence?"}
validate -->|Yes| answer["Grounded answer<br/>with resolved citations"]
validate -->|No| abstain["Explicit abstention"]
end
ui --> question
answer --> ui
abstain --> ui
store --> retrieve
retrieve --> trace["Retrieval, model,<br/>tokens and latency"]
generator --> trace
trace --> answer
classDef source fill:#eef2ff,stroke:#6366f1,color:#1e1b4b
classDef process fill:#ecfeff,stroke:#0891b2,color:#164e63
classDef data fill:#f0fdf4,stroke:#16a34a,color:#14532d
classDef decision fill:#fff7ed,stroke:#ea580c,color:#7c2d12
classDef outcome fill:#fdf2f8,stroke:#db2777,color:#831843
class ui,docs,question source
class parser,embed,qembed,retrieve,generator,trace process
class chunks,store,evidence data
class validate decision
class answer,abstain outcome
The generator is allowed to propose an answer and cite passage IDs, but it never creates trusted citation metadata. The application treats the retrieved passage IDs as an allowlist and resolves the final source titles, rule references, and URLs itself.
flowchart LR
question["Question"] --> retrieval["Retrieve top-k passages"]
store[("pgvector")] --> retrieval
retrieval --> evidence["Trusted passages<br/>+ chunk IDs"]
evidence -->|"question + evidence"| llm["LLM<br/>untrusted draft"]
llm -->|"answer + proposed IDs"| verifier{"Every cited ID<br/>was retrieved?"}
evidence -->|"allowlist"| verifier
verifier -->|"Yes"| citations["Server-resolved<br/>citations"]
verifier -->|"No"| abstention["Explicit abstention"]
classDef trusted fill:#e8f3e8,stroke:#176440,color:#14211c
classDef untrusted fill:#f3dfcf,stroke:#a84b1d,color:#54200f
classDef decision fill:#fff5c2,stroke:#9b7a00,color:#3b3100
class question,retrieval,store,evidence,citations,abstention trusted
class llm untrusted
class verifier decision
The components will communicate through small project-owned interfaces. Provider and framework integrations—including LangChain—will remain replaceable adapters.
The baseline performs exact nearest-neighbor search. Approximate vector indexes are intentionally deferred until corpus size and measured latency justify their recall trade-off.
The initial corpus contains original explanations of selected rules in effect for the 2026 NFL season. Each document records its season, official rule references, and authoritative source URL. The official rulebook is linked rather than redistributed.
The assistant is designed to explain NFL playing rules. Team news, player statistics, fantasy advice, college rules, and live officiating decisions are outside its scope.
- Evaluate retrieval separately from answer generation.
- Prefer explicit evidence and abstention over plausible unsupported answers.
- Add complexity only when a benchmark demonstrates its value.
- Keep ingestion reproducible and the evaluation corpus version-controlled.
- Track quality, latency, and inference usage together.