Repository navigation
enhancement: streaming fact-log iterator with full history (tx, asserted, valid bounds) and coarse filters #430
Description
Motivation
An offline migration (project-minigraf/temporal_reasoning#378) needs to read every fact version in a graph, retracted ones included, in transaction order, without going through Datalog. Today history is reachable only per transaction via :as-of N, and a full-graph Datalog scan on the server's thread has starved an active ingestion for over 20 minutes; there is no API that yields the log as a stream.
Proposal
A read-side API on Minigraf (usable in the read-only mode requested separately):
pub struct FactRecord { entity, attribute, value, tx: u64, asserted: bool, valid_from, valid_to }
pub fn iter_facts(&self, filter: &FactFilter) -> impl Iterator<Item = Result<FactRecord>>- Order: transaction order, then insertion order within a transaction, so a consumer can rebuild the log and keep
:as-of Nmeaning the same thing for facts it copies unchanged. - Filters (all optional, ANDed): entity ident prefix (
:lineage-marker/,:ingestion/), attribute set,txrange, entity-type. A migration that only rewrites two namespaces must not pay to decode the 90% it will copy unchanged; when a filter is present the iterator should also expose the complementary "untouched" ranges so a caller can copy them page-wise (see the writer issue). - Streaming: bounded memory regardless of graph size; no materialised result set, so
max_resultsdoes not apply. - Exposed through the FFI as a cursor (
next_batch(n)), since uniffi has no iterator type.
This also gives the temporal_reasoning FTS sidecar a rebuild path that does not depend on the entity index (cf. minigraf #370).
Activity
Merge pull request #463 from project-minigraf/feat/430-fact-log
adityamukho commentedon Oct 7, 2026
Done. Core:
Minigraf::fact_log(&FactFilter)in #463: tx order or storage order, bounded memory, attribute/attribute-prefix/entity/tx-range filters, and a checkpoint pin while a log is open. Bindings: anext_batch(n)cursor of losslessFactRecords in every binding via #467. Differences from this proposal, decided in the design (docs/superpowers/specs/2026-10-07-fact-log-iterator-design.md): no entity-type filter and no 'untouched ranges' output, because #431's page-range copy was dropped (v8 pages share one dictionary and are not self-contained). Ships in v3.0.0.