Repository navigation
bug: after a kill during an active run the EAVT index silently loses most entities' entries while AEVT stays intact — entity-bound lookups return [] and no rebuild path exists #370
Description
Summary
After a host reboot killed a writer mid-run, the graph reopens without error and every attribute-driven query still returns every fact, but entity-bound lookups return nothing for most entities: the EAVT index has lost their entries while AEVT still holds the facts. Because entity ids are uuid5(NAMESPACE_OID, ident), [:some/ident ?a ?v] is arithmetic that resolves to the right id every time, so the empty result is unambiguously a missing EAVT range, not a resolution problem. checkpoint() cannot repair it — per #315 it rebuilds the four indexes by streaming the existing indexes, so a hole in EAVT is copied forward — and the public API (open, execute, checkpoint) exposes no rebuild from fact pages. Per the audit note on #215, the on-disk checksum covers fact pages only, so this state is silent: no error on open, no error on query, wrong answers.
Environment
minigraf 2.0.0 (via temporal-reasoning[git-ingestion]==0.7.1), Python 3.12.3, Linux 6.8, ext4 on NVMe. Graph: 2.29 GB, ~29,700 :type/commit entities plus ~1.5M code entities (ArangoDB origin/4.0 history, forward walk).
Trigger
2026-09-06 ~10:06 local: host reboot while a temporal_reasoning ingestion run was actively transacting and checkpointing (graph mtime 10:05:28, .wal sidecar and the FTS sidecar's SQLite WAL both left un-checkpointed). On reboot the graph opened cleanly, total_ingested matched the pre-reboot count (29,311), and no error was ever reported. The earlier hard kills in this graph's life (2026-09-03 SIGKILL of a wedged process with no write in flight, 2026-09-04 clean shutdowns) did not produce the symptom — or at least no entity-bound lookup on an untouched entity was made afterwards to notice.
Evidence
Same entity id, two paths, same live database (ingestion running in the same process, single handle):
;; attribute-driven: finds the entity and all 7 facts
(query [:find ?e ?a ?v :where [?e :ident ?i] [(= ?i ":decision/build-files-editable")] [?e ?a ?v]])
→ 7 rows, entity 11569db1-e79d-58d8-b09c-7ad7b0c53981
;; entity-bound, by ident keyword: empty
(query [:find ?a ?v :where [:decision/build-files-editable ?a ?v]])
→ []
;; entity-bound, by the same UUID literal: empty
(query [:find ?a ?v :where [#uuid "11569db1-e79d-58d8-b09c-7ad7b0c53981" ?a ?v]])
→ []
uuid.uuid5(uuid.NAMESPACE_OID, ":decision/build-files-editable") = 11569db1-e79d-58d8-b09c-7ad7b0c53981, confirming the id the ident resolves to is the id AEVT reports. Verified the same way for :ingestion/watermark → e4aa5fe8-4621-533b-94b9-2833a380adde.
Sample of nine entities, all with facts reachable through AEVT, probed by [#uuid "…" ?a ?v]:
| entity | age / origin | EAVT facts |
|---|---|---|
:commit/dc70c6f4d294 (2012 commit) |
written 2026-09-03 | 0 (AEVT: 8) |
:commit/98ff1675d09d (2026 commit) |
written 2026-09-03 | 0 |
:decision/build-files-editable |
written 2026-09-03 | 0 (AEVT: 7) |
:constraint/faiss-default-build-flags |
written 2026-09-03 | 0 |
:module/…/OptimizeJoinOrder.cpp |
written 2026-09-03 | 0 |
:module/…/MatchBuilder.cpp |
written 2026-09-03 | 0 |
:module/…/tests/…/aql-match-statement.js |
written 2026-09-03, untouched since | 21 |
:module/…/Ast.cpp |
rewritten today (new facts) | 484 |
:module/…/ExecutionPlan.cpp |
rewritten today | 501 |
So six of nine are invisible to entity-bound queries; entities the post-reboot process writes to become visible (their new EAVT entries are added), the rest stay missing. Aggregate queries are unaffected: [:find (count-distinct ?e) :where [?e :entity-type :type/module] [?e :path ?p]] = 10,090 both with and without the second clause, which is why the damage went unnoticed for a day.
Consequences observed downstream
- Silent wrong answers for every
[:ident ...]-shaped lookup, the primary access pattern for a graph keyed by idents. - Ingestion collapse.
temporal_reasoningissues one entity-bound point query per candidate entity per commit (_lineage_is_provisional). Against the damaged index those go from ~13 s/commit total to 1–60 min/commit, CPU-bound in the FFI call (six of six py-spy samples insideminigraf_ffi.executefrom that function), with no disk I/O. Reported separately on the temporal_reasoning tracker. - No recovery path. Reopening (three fresh processes),
checkpoint()(dozens since), and clean close/open cycles did not restore the entries. The documented recovery downstream is a full re-ingest into a fresh file.
Asks
- A rebuild-indexes-from-fact-pages operation (or automatic detection + rebuild on open), since fact pages are intact and test: add index corruption recovery and rebuild fixtures #215's audit says the checksum does not cover index pages.
- Consider whether SIGKILL during an active transact has a ~26% chance of permanently corrupting the entire database (header checksum, no recovery path) #308's fix covers the "kill during
save()index rebuild" window: a partially written index tree with a valid header would produce exactly this — a graph that opens fine and answers scans correctly while entity ranges are gone. - Ideally an integrity check (
(verify)or on-open sampling) that compares AEVT and EAVT cardinalities, so this state raises an error instead of returning[].
I can supply the 2.29 GB file for analysis on request, or run any probe against it; it is otherwise being retired in favour of a re-ingest.
Activity
adityamukho commentedon Sep 7, 2026
Downstream consequence and the per-ident probe pattern that amplifies it: project-minigraf/temporal_reasoning#336.
fix: derive FactRefs from on-disk layout when rebuilding indexes on o…
adityamukho commentedon Sep 21, 2026
Root cause of the lost entity lookups fixed in #372 (index rebuild on open derived FactRefs by re-packing all facts contiguously, which doesn't match the per-checkpoint page layout). Remaining asks are tracked in #373 (integrity check + public rebuild) and #374 (crash-atomic save()). Files already rebuilt with the bad refs need a rebuild from fact pages (#373) or a re-ingest.
revert: move file format v8 (#377) from main to the v3 branch (#378)
release: v2.0.1 — fix index rebuild on open (#370)
docs(test-coverage): keep the #370 index-rebuild regression test in s…