This is a guided tour of the code: what runs during ingest, query and eval, which data types move between the steps, what each score means, and where files live. Class and method names refer to src/mini_rag. The reasons behind each choice are in decisions.md.
flowchart TD
subgraph S1["1. Ingest (run once)"]
files["Text files<br/>20 Wikipedia pages"] --> loader["Loader"]
loader --> chunker["Chunker<br/>500 chars, 100 overlap"]
chunker --> dense["Dense index<br/>bge-small vectors, saved"]
end
subgraph S2["2. Query (per question)"]
question["Question"] --> bm25["BM25 search<br/>keyword match"]
question --> densesearch["Dense search<br/>meaning match"]
bm25 --> fuser["RankFuser<br/>reciprocal rank fusion"]
densesearch --> fuser
fuser --> answerer["Answerer<br/>dense score gate"]
answerer --> answer["Answer<br/>or not found"]
end
subgraph S3["3. Eval (measures the search step)"]
testset["Question set"] --> evaluator["Evaluator"]
evaluator --> metrics["Metrics<br/>hit@k, recall@k, MRR"]
metrics --> results["results.md"]
end
config["Config<br/>mini-rag.toml"] --> S1
config --> S2
dense -.-> bm25
dense -.-> densesearch
classDef done fill:#d4edda,stroke:#2e7d32,color:#1b4d20
classDef todo fill:#fff3cd,stroke:#b58900,color:#5c4400
class config,files,loader,chunker,dense,question,bm25,densesearch,fuser,answerer,answer done
class testset,evaluator,metrics,results done
Legend: green means built and merged, yellow means not built yet. The dotted lines show that both searches read the saved index. The RankFuser step runs in hybrid mode only: dense mode skips BM25 search, and bm25 mode skips dense search (D31).
Before any command runs, Cli.load_config() finds --config among the arguments and ConfigLoader.load() reads that TOML file, or mini-rag.toml in the working directory when the option is left out, into a frozen RagConfig. A key left out of the file keeps its built-in default, a missing default file means all defaults, and a wrong type, or a key, section or kind outside the schema, is an error that names the key. Cli.build_parser() then uses the config's values as the option defaults, so the precedence is, highest first: the command-line option, the config file, the built-in default.
ComponentFactory turns the [embedder] and [generator] sections into objects. embedder() returns a FastEmbedEmbedder after checking the model name against fastembed's list of supported models, or for kind custom imports the named Embedder subclass and passes [embedder.options] as keyword arguments. generator() returns an LlmClient (kind anthropic), an OpenAiCompatibleClient (kind openai_compatible), None (kind none) or a custom Generator subclass. Embedder and Generator are the abstract base classes in embedder.py and generator.py. Tests pass an embedder or a client to Cli directly, which skips the factory.
uv run mini-rag ingest calls Cli.ingest, which runs these steps in order:
Chunker(i_size, i_overlap)validates the chunk settings first, so bad values fail before any work is done.Loader(s_corpus_dir, t_extensions).load()reads every non-empty.txtfile indata/corpus(and.mdfiles when[corpus] extensionsnames them) in file name order and returns oneDocumentper file. Line endings are normalised to\n.Chunker.chunk_documents()cuts every document intoChunkobjects of 500 characters, each starting 400 characters after the last. On the included corpus that's 20 documents and 1,741 chunks.DenseIndex.build()embeds all chunk texts through the configured embedder'sembed(), by defaultFastEmbedEmbedder, and stores them as L2-normalised float32 rows inarr_vectors.DenseIndex.save()writesvectors.npyandchunks.jsoninto.cache/index.
Finally, Cli.ingest prints the counts, the settings, the index folder and the elapsed time. A missing corpus folder or an invalid setting prints an error and exits with code 2.
uv run mini-rag query "<question>" calls Cli.query:
DenseIndex.load()reads.cache/indexand checks the 2-D shape, the row count and the saved embedder name against the configured embedder'ss_name. A different name stops the query with a message that asks for a newingest.ComponentFactory.retriever()builds the retriever for[retrieval] mode, or for--modewhen it's given. In hybrid and bm25 modesBm25Index.build()rebuilds the keyword index from the loaded chunks. It isn't saved separately. Dense mode builds no BM25 index.Answerer(f_threshold, b_gate_on_dense=...)is created before searching, so an invalid--thresholdfails early.b_gate_on_denseisFalsein bm25 mode andTrueotherwise.HybridRetriever.search(question, top_k)asks each index formax(20, top_k)candidates:Bm25Index.search()tokenises the question withTokenizerand keeps hits with a positive score, andDenseIndex.search()embeds the question withquery_embed()and ranks every chunk by cosine score. In dense modeDenseOnlyRetriever.search()returns the toptop_kdense hits instead, and in bm25 modeBm25OnlyRetriever.search()returns the toptop_kBM25 hits without embedding the question. Both skip step 5 and leave the fused scoreNone.RankFuser.fuse()merges the two lists with weighted reciprocal rank fusion. Each list has a weight,f_bm25_weightandf_dense_weightonHybridRetriever, and a chunk's fused score is the sum of weight / (60 + rank) over the lists that contain it. Both weights default to 1.0, which gives plain reciprocal rank fusion.HybridRetrieverthen attaches each chunk's own dense and BM25 score and keeps the firsttop_khits asHybridHitobjects.Cli.print_hit()prints every hit with its marker and its three scores, with-for a missing score. In bm25 modeCli.querythen prints a note that the threshold isn't applied.Cli.answer()picks the answerer. With--generate,Cli.make_client()asksComponentFactory.generator()for the configured client. For kindanthropicit first checks thatANTHROPIC_API_KEYis set, and for kindnone, the default, it prints that no generator is configured and returns no client. Without a client, or when the client fails, the extractive answerer is used and a note is printed.Answerer.answer()applies the gate inis_found(). If the best dense score among the hits is below the threshold, the result is "Not found in the corpus." In bm25 modeis_found()skips the threshold and returnsTruewhen at least one hit exists. Otherwiseformat_sources()builds the extractive answer, orLlmClient.generate()writes one from the same sources.Cli.print_answer()prints the answer text and a line such asfound: yes, written by a model: no.
uv run mini-rag eval calls Cli.eval:
QuestionLoader.load()readseval/questions.jsonand returns oneQuestionper entry. A missing file, a malformed entry or a duplicate id prints an error and exits with code 2.Loader.load()reads the corpus, andQuestionValidator.validate()checks every question against it. If there are problems, the command prints up to 10 of them and exits with code 2.Evaluator.run()asksEvalIndexStore.get_indexes()for the indexes of each chunk size. The store loads the dense index from.cache/eval/size_<size>_overlap_<overlap>whenmeta.jsonmatches the size, overlap, model name and corpus fingerprint. Otherwise it chunks, embeds and saves. It rebuilds the BM25 index from the chunks every time.- For each size,
Evaluator.run_size()wraps the indexes inBm25Method,DenseMethodand oneHybridMethodperFusionSetting. All three implementRetrievalMethod.rank(). A plain run has a single fusion setting, the defaults ofHybridRetriever. - Each method ranks every answerable question to depth 10.
Evaluator.outcome()callsRelevanceJudge.is_relevant()on each returned chunk, which is true when the chunk contains the question'smust_containphrase, and records aQuestionOutcome. HybridMethod.hits()then returns the top 5 hybrid hits for every question, answerable or not, andEvaluator.best_dense()keeps the highest dense score among them.ThresholdSweeper.sweep()turns those best scores into oneSweepRowper threshold from 0.40 to 0.80.ReportWriter.console_text()gives the text thatCli.evalprints (Table 1 and Table 3), andReportWriter.write()saves all tables toeval/results.md.ReportTablesbuilds the tables and usesMetricCalculatorfor hit@k, recall@k and MRR.
With --build-only, Cli.build_only runs Evaluator.prepare() for each size and prints the chunk count, whether the index was built or loaded, and the elapsed seconds. It loads no questions and writes no report.
Four more commands help a user build a question set. eval-init calls QuestionStarter.write(), which writes a starter file and refuses to replace one without --force. eval-suggest calls SentenceSuggester.suggest(), which picks sentences with a digit from the corpus, at most 5 per document. eval-check runs QuestionValidator.validate(); with --lenient it skips the set-level rules. calibrate builds a Calibrator, which loads or builds the index of the configured chunk size through EvalIndexStore, takes the best dense score of each question from the hybrid hits, and sweeps thresholds across the measured range with ThresholdSweeper. EvalKit prints the results for all four.
With --fusion-grid, FusionGrid.settings() supplies 12 fusion settings: k of 5, 10, 20 or 60, each with a BM25 weight of 1.0, 0.5 or 0.25 and a dense weight of 1.0. Cli.eval prints whether each index was loaded or built. Evaluator.run_fusion() then evaluates hybrid once per setting and stores a FusionResult in the SizeResult, while BM25 and dense are still ranked once per size. The tables in eval/results.md keep using the default setting, and FusionGridWriter.write() saves one row per setting to fusion_grid.md in the report's folder.
The sweep uses the gate's own rule. Answerer.is_found() takes the highest non-None dense score among the hits and compares it with the threshold. Evaluator.best_dense() stores that same maximum for the top 5 hybrid hits of each question, and ThresholdSweeper.is_found() counts a question as found when the stored score is at least the threshold. A test runs both on the same hits and checks that they agree. The score belongs to whichever of the five hits has the highest cosine, which need not be the hit ranked first. The design and the measured results are in decisions.md, D15 and D19.
All are frozen dataclasses except EvalResult and the SizeResult objects it holds.
| Type | Fields | Made by | Meaning |
|---|---|---|---|
Document |
s_doc_id: str, s_text: str |
Loader |
One corpus file. The id is the file name stem, such as tokamak. |
Chunk |
s_doc_id: str, i_index: int, i_start: int, s_text: str |
Chunker |
A slice of a document. i_index is its position within the document, and i_start is its character offset. Its identity is (s_doc_id, i_index). |
SearchHit |
obj_chunk: Chunk, f_score: float |
Bm25Index, DenseIndex |
One result from a single retriever, with that retriever's score. |
HybridHit |
obj_chunk: Chunk, f_rrf_score: float | None, f_dense_score: float | None, f_bm25_score: float | None |
HybridRetriever, DenseOnlyRetriever, Bm25OnlyRetriever |
One result. A score is None when that retriever didn't return the chunk among its candidates. f_rrf_score is None in dense and bm25 modes, which fuse nothing. |
Answer |
s_text: str, l_sources: list[Chunk], b_found: bool, b_generated: bool |
Answerer |
The final answer. l_sources is empty when nothing was found, and b_generated is True only when a model wrote s_text. |
Question |
s_id: str, s_text: str, b_answerable: bool, s_source_doc: str, s_must_contain: str, t_absent_terms: tuple[str, ...], s_topic: str |
QuestionLoader |
One eval question. Answerable questions fill s_source_doc and s_must_contain; unanswerable ones fill t_absent_terms and s_topic. Unused fields are "" or an empty tuple. |
QuestionOutcome |
s_id: str, i_first_rank: int | None, i_total_relevant: int, d_found_at_k: dict[int, int] |
Evaluator |
How one method did on one answerable question. i_first_rank is 1-based and None when no relevant chunk was in the top 10. d_found_at_k counts the relevant chunks in the top 1, 3 and 5. |
SweepRow |
f_threshold: float, i_answerable_rejected: int, i_answerable_total: int, i_unanswerable_accepted: int, i_unanswerable_total: int |
ThresholdSweeper |
The two kinds of gate error at one threshold. |
EvalResult |
s_corpus_fingerprint: str, s_model_name: str, i_overlap: int, i_depth: int, i_gate_top_k: int, l_size_results: list[SizeResult] |
Evaluator |
One eval run. Each SizeResult holds a chunk size's chunk count, the outcomes per method, the best dense score per question, the sweep rows and one FusionResult per fusion setting. |
FusionSetting |
i_k: int, f_bm25_weight: float, f_dense_weight: float |
FusionGrid, FusionSetting.default() |
One combination of fusion parameters. s_label gives its report name, such as k=60 w=1.0/1.0. |
FusionResult |
obj_setting: FusionSetting, l_outcomes: list[QuestionOutcome], d_best_dense: dict[str, float | None], l_sweep_rows: list[SweepRow] |
Evaluator |
Hybrid's outcomes, gate scores and sweep under one fusion setting at one chunk size. |
| Score | Range | What it means | Comparable with |
|---|---|---|---|
| BM25 | 0 upwards (non-positive hits are dropped) | Keyword overlap, weighted by how rare each word i 4F61 s and adjusted for chunk length. | Other BM25 scores for the same question only. The scale changes with the question's words. |
| Dense cosine | -1 to 1 | Similarity in meaning between the question vector and the chunk vector. | Other cosine scores from the same model. The range is fixed, so one threshold can be applied to every question, which is why the answer gate uses it. |
| Fused RRF | above 0, at most 2/61 (0.0328 to four decimals) | The sum of weight / (60 + rank) over the retrievers that returned the chunk, with both weights at their default of 1.0. Only the ranks matter. | Other fused scores for the same question only. It can't be compared with BM25 or cosine scores. |
All paths are relative to the project root.
| Path | Contents | In git |
|---|---|---|
data/corpus/ |
The 20 article texts and SOURCES.md with each article's revision id and SHA-256 |
yes |
assets/readme/ |
banner.svg and badges.svg, the two images at the top of the README, checked by tests/test_readme_assets.py |
yes |
data/corpus_titles.txt |
The article titles that CorpusDownloader fetches |
yes |
.cache/index/ |
vectors.npy (1,741 × 384 float32) and chunks.json (chunks plus model name), written by ingest |
no |
.cache/eval/ |
One folder per chunk setting, such as size_500_overlap_100, with vectors.npy, chunks.json and meta.json, written by eval |
no |
.cache/fastembed/ |
The quantised embedding model, in models--Qdrant--bge-small-en-v1.5-onnx-Q, 64.1 MB (67,181,330 bytes) |
no |
.cache/huggingface/ |
Hugging Face hub files, kept here by HF_HOME |
no |
.cache/pytest-tmp/ |
Temporary folders made by tests | no |
eval/questions.json |
The 25 answerable and 6 unanswerable eval questions | yes |
eval/results.md |
The report of the last eval run, written by eval |
yes |
eval/fusion_grid.md |
One table per chunk size with a row per fusion setting, written by eval --fusion-grid |
yes |
eval/my_questions.json |
A starter question file, written by eval-init |
no, until you add it |
mini-rag.toml |
The config file every command reads; the shipped one reproduces the included example | yes |
examples/custom_embedder.py |
A minimal Embedder subclass to copy |
yes |
examples/custom.toml |
A config that uses that embedder, with its own index and eval cache folders | yes |
.cache/index_custom/, .cache/eval_custom/ |
The index and the eval cache written with examples/custom.toml |
no |
docs/make-your-own.md |
How to swap the corpus, the models and the question set | yes |
scripts/env.ps1 |
Sets FASTEMBED_CACHE_PATH and HF_HOME for the session |
yes |
src/mini_rag/ |
The package, one class per file | yes |
tests/ |
The test suite and its fakes | yes |
The question "Who won the 1998 football World Cup?" has no answer in the corpus, since none of the 20 articles mentions the World Cup. Running it with the defaults (--top-k 5, --threshold 0.55) gives:
1. [nuclear_safety_and_security#82] dense 0.502 bm25 14.754 fused 0.0313
2. [chernobyl_disaster#158] dense 0.495 bm25 8.279 fused 0.0288
3. [chernobyl_disaster#145] dense 0.528 bm25 - fused 0.0164
4. [tokamak#94] dense - bm25 12.990 fused 0.0161
5. [chernobyl_disaster#6] dense 0.520 bm25 - fused 0.0161
Here's how to read it.
- BM25. The tokens are
who,won,the,1998,football,worldandcup. Hit 1 contains "who", "1998" and "world" (from a passage on a 1998 coolant loss at a French reactor). Hit 4 contains "won" and "world". Both score high on words that have nothing to do with football. - Dense. Hit 3 has the highest cosine score in the whole index, 0.528, but BM25 didn't return it, so its BM25 column shows
-. - Fusion. Hit 3 is first in the dense ranking alone, so its fused score is 1/61, which is 0.0164 to four decimals. Hits 4 and 5 are each second in one ranking, so both get 1/62, which is 0.0161 to four decimals. The tie keeps BM25's hit first, which is why
tokamak#94comes beforechernobyl_disaster#6. Hit 1 ranks first because both retrievers returned it. - Gate. The top fused hit has a dense score of 0.502, but the gate reads the best dense score among all five hits, 0.528. That's below 0.55, so the output is "Not found in the corpus." with
found: no, which is correct.
With --threshold 0.5 the same hits come back, 0.528 is at least the threshold, and the five chunk texts are printed as the extractive answer. That's a wrong acceptance, and 0.5 was the default until the threshold sweep in eval/results.md counted both kinds of error at each threshold (see decisions.md, D23).