Repository navigation
Graph files are 42-70% larger on v2.0.2 than v2.0.0 for identical facts when written over many checkpoints #428
Description
Summary
For the same facts, a graph written over many checkpoints comes out 42–70% larger on v2.0.2 than on v2.0.0. With a single checkpoint the file is byte-for-byte the same size on both versions, so the growth comes from repeated checkpoints. The only checkpoint change between those tags is #315's incremental index rebuild (12b1ec3, #402), so that is the likely cause. Not verified: that the re-split leaves index leaves partly filled.
Once written, the file is stable: further open/checkpoint/close cycles don't change it. So it is a density difference, not unbounded growth. No correctness impact was observed: identical fact sets and query results.
Standalone repro
import os, sys, json, tempfile
from minigraf import MiniGrafDb
p = os.path.join(tempfile.mkdtemp(), "g.graph")
for b in range(400):
db = MiniGrafDb.open(p)
facts = " ".join(f'[:e/b{b}-{i} :name "n{b}-{i}"] [:e/b{b}-{i} :n {b*10+i}]' for i in range(10))
db.execute(f"(transact [{facts}])")
del db # drop runs a checkpoint
db = MiniGrafDb.open(p)
print(len(json.loads(db.execute("(query [:find ?e ?a ?v :where [?e ?a ?v]])"))["results"]))
db.checkpoint(); del db
print(os.path.getsize(p)) | v2.0.0 | v2.0.2 | |
|---|---|---|
| 8,000 facts, 400 open/transact/close cycles | 3,276,800 B | 4,657,152 B (+42%) |
same, plus an explicit checkpoint() every cycle |
3,276,800 B | 4,657,152 B |
| same 8,000 facts in ONE transact + one checkpoint | 2,039,808 B | 2,039,808 B |
Downstream measurement
temporal_reasoning's full-history ingestion of its own repo (1,021 commits, ~1,200 handle drops and ~85 explicit checkpoints per run) wrote identical graphs on both versions: 38,643 facts, fact_audit divergence 0, in two runs each. The files differed:
- 352 MB on 2.0.0 vs 597 MB on 2.0.2 (+70%), in both repetitions.
- 5.0 MB vs 8.2 MB on a 151-commit slice.
Data: project-minigraf/temporal_reasoning evals/at_scale/results/239-minigraf-2.0.2-ab.json (the graph_bytes field).
Why it may matter beyond disk
- A checkpoint on v2.x still copies every index page. If a larger file means more index pages, then each later checkpoint and each checkpoint-on-drop costs more (feat(ffi): expose OpenOptions over UniFFI — the close-time checkpoint cannot be suppressed from Python #322 context).
- Downstream, handle-drop cost on 2.0.2 was too noisy across two runs (59% spread) to say whether perf: checkpoint() is O(graph size) and flat in dirty bytes — it rebuilds all four covering indexes every call #315's speedup is net positive after the size growth. Worth measuring here with the repro above.
Environment
- Linux x86_64, Python 3.14, PyPI wheels
minigraf==2.0.0andminigraf==2.0.2, in twin venvs with identicalpip freezeapart from minigraf.
Activity
adityamukho commentedon Oct 7, 2026
Re-measured with the repro ported to Rust (8,000 facts = 400 open/transact/close cycles of 20 facts; then 5 extra open/checkpoint/close cycles to check stability).
v3at 8346bd1 (format v8, copy-on-write checkpoints from #451) vsmainat 596bb59 (v2.x, format v7).checkpoint()mainreproduces the reported 4,657,152 B exactly. On v3 the many-checkpoint file is 3.3× smaller than on v2.0.2 and 2.3× smaller than the v2.0.0 figure in the report (3,276,800 B).Scaling on v3 (B/fact, many cycles vs one checkpoint): 2k facts 203 vs 131 · 8k 175 vs 120 · 16k 172 vs 120 · 32k 171 vs 135. The per-fact cost stays flat as cycles grow, so the remaining ~1.3–1.45× overhead is bounded.
verify()is clean in every case. At 8k facts it checks 319 live pages out of 341 in the file (22 free-list pages), against 233/235 for the single-checkpoint file. Nearly all of the gap is in live pages that incremental copy-on-write writes partly filled, not leaked space. Further open/checkpoint/close cycles don't change the size on either branch.Closing: the regression is resolved in v3.0.0 by the v8 copy-on-write checkpoint (#434/#374, PR #451). v2.x will not get a density fix (patch releases are data-integrity and security only).