8000 minigraf/CLAUDE.md at v3 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]
Skip to content
8000

CLAUDE.md

Latest commit

 

History

History
316 lines (242 loc) · 38.8 KB

CLAUDE.md

File metadata and controls

316 lines (242 loc) · 38.8 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

Minigraf is a tiny, portable bi-temporal graph database with Datalog queries written in Rust. Designed as the embedded graph memory layer for AI agents, mobile apps, and the browser — built on the SQLite philosophy: embedded, single-file, reliable, with time travel.

See ROADMAP.md for planned work and current direction. See CHANGELOG.md for completed-release history. The #836 benchmark audit records the 156 unselected native IDs and per-shape manual dispositions; it changes no runtime behavior or test counts.

The #838 manual migration sweep covers native v7 migration through 200k facts and Chromium/Firefox/WebKit through 20k facts, including bounded post-migration cursor churn. It changes neither nightly coverage nor the Rust test inventory; larger browser cases remain excluded.

The #840 regression-history follow-up verifies numeric comparisons for all 427 native and 26 8000 binding IDs, at least three independent v3 runs per ID, preserved history after main advances, actual scheduled routes and isolated live notification delivery/cleanup. Fixes #853/#854 are merged; checksummed evidence outlives Actions retention. This changes no database runtime behavior or Rust test counts. Additional JVM/WASI and qualified mobile nightly lanes are tracked separately in #858.

The #842 semantic permutation map ties the excluded numeric, bind, join and forced-plan permutations to 128 correctness tests and documents the nightly controls' attribution limits. It changes no code, tests or nightly routes.

The #843 failure-path map ties the failure paths excluded from nightly timing to 138 correctness tests with their error and data invariants. It changes no code, tests or nightly routes; the heavy WAL replay test gap is #865.

The #841 manual shape sweep records native sort/window, set, UUID, recursive-control and sustained-reader experiments, including rejected inputs, capped attempts and attribution limits. Its standalone harness and disposable diagnostic probes do not change the library test inventory, production code or nightly routes.

Core Philosophy - CRITICAL

Before implementing ANY feature or change, you MUST assess it against the design philosophy in PHILOSOPHY.md.

Minigraf follows the "SQLite for bi-temporal graph databases" philosophy:

  • Zero-configuration - No setup, no config files, just works
  • Embedded-first - Library not server, in-process execution
  • Single-file database - One portable .graph file
  • Self-contained - Minimal dependencies, small binary (~1.2MB budget as of #277, 1,240 KiB enforced on v3 since #870; see PHILOSOPHY.md §4)
  • Cross-platform - Native, WASM, mobile, embedded
  • Reliability over features - Do less, do it perfectly
  • Bi-temporal first-class - Time travel is a core feature, not addon
  • Datalog queries - Simpler, more powerful for graphs than SQL/GQL
  • Stability - Backwards compatibility, stable file format
  • Long-term support - Decades-long commitment

Philosophy Compliance Check

CRITICAL INSTRUCTION: When the user requests a feature or change, you MUST:

  1. First, assess whether it aligns with the philosophy in PHILOSOPHY.md
  2. If it violates the philosophy, WARN the user BEFORE implementing:
    • Explain which principle(s) it violates
    • Explain why it's problematic
    • Suggest alternatives that align with the philosophy
    • Ask for explicit confirmation to proceed despite the violation
  3. If it aligns, proceed with implementation

Examples of philosophy violations to warn about:

  • Adding client-server architecture (violates embedded-first)
  • Requiring external services or complex setup (violates zero-configuration)
  • Large dependencies that bloat binary size (violates self-contained)
  • Features only useful for distributed systems (violates target use cases)
  • Breaking changes to API or file format (violates stability)
  • Complex features before basics are reliable (violates reliability-first)
  • Multi-file storage (violates single-file philosophy)

Your response format when detecting a violation:

⚠️ PHILOSOPHY WARNING ⚠️

The requested feature/change may violate Minigraf's core philosophy:

**Violated Principle**: [principle name from PHILOSOPHY.md]

**Why this is problematic**: [explanation]

**Philosophy-aligned alternatives**:
- [alternative 1]
- [alternative 2]

Do you want to proceed anyway? This would be a deviation from the "SQLite for graph databases" philosophy.

See PHILOSOPHY.md for complete design principles and decision framework.

Build and Run Commands

# Build
cargo build
cargo build --release

# Run the REPL
cargo run

# Run tests
cargo test

# Run specific test suite
cargo test --test bitemporal_test -- --nocapture
cargo test --test window_functions_test -- --nocapture
cargo test --test complex_queries_test -- --nocapture
cargo test --test recursive_rules_test -- --nocapture
cargo test --test concurrency_test

# Kani proofs of key encoders and pure scalar decoders (#627, #693):
# 29 harnesses, ~2 minutes warm and 7.6 GiB peak aggregate CBMC RSS at -j 4.
# Measure any new harness on the bench machine before adding it to CI.
# Reader error paths and variable-length strings remain outside Kani's budget.
cargo kani --lib -j 4 --output-format terse -Z unstable-options --harness-timeout 10m

# Run examples
cargo run --example embedded
cargo run --example file_storage

# Performance envelope (#394): latency/RSS at 100K–10M facts; opt-level 3 profile
cargo run --profile envelope --example perf_envelope -- --scales 100k,1m

# Run demo scripts
cargo run < demos/demo_commands.txt
cargo run < demos/demo_recursive.txt
cargo run < demos/demo_bitemporal.txt
cargo run < demos/demo_negation.txt

Architecture

Module Structure

  1. src/graph/ — EAV fact store with bi-temporal support

    • types.rs: Fact, Value, EntityId, TxId, VALID_TIME_FOREVER
    • storage.rs: FactStorage — in-memory store, snapshot() → Snapshot (committed reader + pin + pending prefix, and a WriteTransaction's staged facts read with the pending ones as one store, #550; every query/cursor/log read goes through one), transact_batch, retract, get_facts_as_of, get_facts_valid_at, net_asserted_facts, get_facts_by_entity_attribute_indexed; Snapshot::has_pending probes only its captured pending/staged source range for input-order proofs (#641)

  2. src/storage/ — Persistence layer

    • mod.rs: StorageBackend trait, LegacyHeaderV7 (v7 migration only), CommittedReader trait
    • backend/file.rs: Single .graph file backend (4KB pages, cross-platform)
    • backend/memory.rs: In-memory backend for testing
    • backend/fault_inject.rs: FaultInjectingBackend — injects I/O errors for durability tests (test builds only)
    • fault.rs: thread-local fault plan (test builds only, #390) — EIO, torn/short writes, sticky ENOSPC, failed fsync with or without lost writes, at the k-th write or sync of FileBackend, the WAL and sync_parent_dir; driven by src/fault_matrix.rs
    • index.rs: pending (uncheckpointed) EAVT/AEVT keys over UUIDs and strings, encode_value; index/verification.rs: Kani proofs of encode_value (#[cfg(kani)], #627)
    • keys.rs: byte-comparable v8 keys (FDB integers, value tags, tx↓, FOREVER), DICT keys, MAX_VALUE_BYTES, MAX_IDENT_BYTES; sets (#316): tag 0x08, element encoding (set_elements: each element's key encoding, sorted) inline up to 64 bytes, else LongSet in a value page, decode_set, set_bytes_bound (the WAL's id-free size bound); pure scalar decoders preserve INT-049 errors (#693). keys/verification.rs proves encoder order and scalar decoder round trips/malformed bounds with Kani (#[cfg(kani)], #627, #693); keys/legacy_verification.rs compares the old and new readers in tests. kani.yml runs proofs on key PRs and nightly.
    • node.rs: prefix-compressed leaf and shortest-separator internal node codecs
    • btree.rs: On-disk B+tree over byte keys (build_btree, cow_update (copy-on-write inserts and deletes; an emptied leaf or node leaves its parent, an emptied tree is root 0; #550) and cow_insert, LeafCursor (with seek), prefix_scan, get, MutexStorageBackend)
    • dict.rs: DictReader (id ↔ UUID/ident, tx timestamps, long values; uuids_of walks dense id batches with LeafCursor and uses exact scalar reads for sparse/small batches, #530) and Encoder (checkpoint-time id assignment and key building)
    • value_pages.rs: append-only value pages for strings over 64 bytes and sets whose element encoding is over 64 bytes (#316; DICT dedup compares bytes, long_bytes/find_long_bytes)
    • reader.rs: OnDiskReader — covering reads of committed facts (CommittedReader, KeySource); query scans use the resumable live_keys to decide net-assert on index keys (#379) among the versions a TxBound keeps; live_facts is test-only, and production fact-level net-assert handles pending records (#532, superseded by #551); KeySource::any_key proves raw kind-prefix presence with one leaf seek, including historical/retracted keys (#641); tx_order (cached per reader), last_tx_at, tx_runs over the DICT's transactions (#551)
    • verify.rs: integrity walk for Minigraf::verify (tree order, index digests, DICT, page accounting) and the source choice for rebuild_indexes (#373)
    • meta.rs / page.rs / freelist.rs: meta pages A/B, the common page header and allocator (skips quarantined ids), free-list chain
    • pins.rs: Pins / GenerationPin — generations held by open snapshots and the quarantine of pages checkpoints freed after them (#543)
    • cache.rs: LRU page cache (PageCache, default 256 pages)
    • dir_sync.rs: sync_parent_dir — fsyncs the parent directory after creating the .graph/WAL or deleting the WAL (no-op off Unix)
    • packed_pages.rs: v7 fact pages, read only for migration
    • persistent_facts.rs: PersistentFactStorage — v8 save/load, auto-migration v7→v8
    • persistent_facts/large_tx.rs: large transactions (#550): a transaction whose staged facts would pass OpenOptions::transaction_chunk_facts (default 65,536, clamped to 1..=1,000,000) on a file database (Minigraf::execute of a larger statement, or a WriteTransaction statement that takes it over) becomes large (WAL guard: the WAL exists, header only, before any page; cache generation bound g + 1; next tx_count reserved, allocated at the commit point; tx_id fixed; the pending store written as the first chunk, so in-transaction reads see the working trees alone), writes each chunk into copy-on-write working trees (per-chunk Encoder over the working DICT, no SharedDictCache; a fact the statement wrote kept once, another window of its triple API-011; an earlier statement's keys of a triple it writes deleted, and the triple recorded in the statement's claims tree; replaced pages: written since the statement's base released, written before it held, g's freed), and commits as a checkpoint (meta g + 1, WAL deleted). In a WriteTransaction each statement has a base (roots, counters, freed length, PageAllocator::begin_base ids): a rejected statement returns to it and the transaction stays usable; staged statements are written as one chunk when it becomes large (their windows checked at its tx_id first). Minigraf::execute's statement error rolls back (cache bound g, written pages evicted, handle dirty so the next checkpoint writes a meta before deleting the WAL); an I/O failure also marks the handle failed (STG-046). A WAL found at open also marks the handle dirty

  3. src/query/datalog/ — Datalog engine

    • parser.rs: EDN/Datalog parser — transact, retract, query, rule, :as-of, :valid-at, not, not-join; resolve_pseudo_sources names each per-fact pseudo-attribute's source (nearest earlier same-entity pattern, lexical scope) and its hidden metadata variable ?#<n>.<attr>, else PRS-081 (#594)
    • parser/statement.rs: transact/retract read one fact at a time (#550): kind_of dispatches on the first two tokens; one pass keeps today's error order (phases 1 tokens … 6 valid windows, the earliest phase's first error, phase 3 in today's check order); read_statement (pass 1) keeps at most TX_CHUNK_FACTS materialised facts and the offset after the last, and returns phase 5–6 errors from Statement::facts/Statement::chunks (one Chunk at a time; pass 2 from that offset for a larger statement); set_tx_chunk_facts for tests (minigraf::__tx_chunk_facts and __statement_facts_high_water under test-hooks). The Parser reads tokens on demand from a Lexer (keep off checks structure without holding it); parse_transact/parse_retract remain under cfg(test) for the differential tests
    • parser/verifier.rs: test-only (not wasm32) bounded-exhaustive check of the parser against tests/grammar/grammar.pest (#643): every 1–3 character string (parse_edn) and every balanced token filler up to MINIGRAF_GRAMMAR_BOUND (default 3) in 17 command templates (parse_datalog_command; 42 tokens); pest rejects ⇒ parser rejects, pest accepts ⇒ parser accepts or raises a SEMANTIC code; today's drift is KNOWN_DRIFT, one entry per cause, removed by its fix (empty: #649, #650, #651 fixed)
    • executor.rs: DatalogExecutor — transact/retract/rule commands, and every query on the engine (open_query, open_rule_query; planner errors after the pseudo-attribute check, then every predicate the query and its rules name, QRY-007); its own query path is gone (#551)
    • matcher.rs: edn_to_value, edn_to_entity_id
    • aggregate.rs: Agg (one aggregate of one group) and Running (a window's running value), used by the engine; exact sums (i128 integers saturated once, fixed-point floats rounded once), value_order (one total order of values for window sorts, peers, rank, min/max and :order-by), row_order (#306), named_vars (a window's tie-break variables), windows (each window's value per row: charged partition indices, in-place total sort, peers) (#547, #765)
    • functions.rs: FunctionRegistry (built-ins and UDFs; Clone shares the closures), Functions — the registry a database shares, RwLock<Arc<FunctionRegistry>>: a query takes a snapshot() when it opens and holds no lock while a UDF runs, so UDFs may call back into the database; registration copies, inserts and swaps (#180). UdfError: a try_register_* closure's failure, QRY-014; eval_expr's EvalError is NoValue (row dropped) or Udf (query fails)
    • evaluator.rs: rule_invocation_to_pattern, the rule limits (DEFAULT_MAX_ITERATIONS = DEFAULT_MAX_DERIVED_FACTS = 1,000,000, #861) and their errors
    • stratification.rs: DependencyGraph, stratify() — negative edges + cycle detection; a positive edge to a rule with not/not-join/or/or-join also raises the caller, off a cycle (#628)
    • rules.rs: RuleRegistry — thread-safe rule management
    • types.rs: EdnValue, Pattern, DatalogQuery, AsOf, ValidAt, Count (:limit/:offset (#306) and :max-derived-facts/:max-results/:query-memory-limit (#870)/:max-iterations (#861) literal or slot), DatalogQuery::order_by (:find elements with Order, #306), DatalogQuery::distinct (default-false bare flag, #493), WhereClause (incl. Not, NotJoin); PseudoAttr enum, AttributeSpec wrapper
    • optimizer.rs: Selectivity-based join reordering on every build (the wasm feature is a deprecated no-op). reorder() gates every reordering site; tests turn it off per thread (set_reorder, minigraf::__reorder under test-hooks, #596); plan_seeded orders a body with its imported variables counted as bound, plan_after orders a rule body's clauses after its delta, each joining on a variable bound before it where one can (#694), and subplans() lets tests force the engine's negation bodies and or branches hashed or correlated (set_subplans, minigraf::__subplans, #546)
    • prepared.rs: BindValue, PreparedQuery — parse-once/execute-many with named $slot bind slots; prepare_query, substitute; Entity and Val(Ref) fit each other's slots, bind mistakes are API-022/023/024 (#181); a :limit $n or :offset $o slot takes Val(Integer(n >= 0)) (resolve_count_slot, #306); limit-keyword slots follow their literal range (resolve_limit_slot, #870); with_max_derived_facts/with_query_memory_limit/with_max_iterations replace the database limits for one statement, ungated (#870, #861)
    • magic_sets.rs: Magic-sets rewriting for demand-driven recursive rule evaluation (not applied to rules with not/not-join)

3a. src/query/engine/ — Streaming query engine (#473), runs every query (#551): plan.rs (the operator tree of every query, or the executor's error code for it: patterns, expression clauses, not/not-join and or/or-join, and rule invocations, read as patterns of their relations (plan_rules, #548; a magic-set head constant is HeadArg::Const), in optimizer::plan order, then the ors in text order, then the deferred nots; an or branch or a not body is planned as a :where is; a clause list with no pattern first starts from Seed; a constant no cell equals is Term::Never, a scan that reads nothing; :as-of resolved when the query opens (Snapshot::tx_bound: a counter bound when tx_id rises with tx_count, else TxBound::Timestamp: committed tx_count ranges and pending tx_id <= t); a per-fact pseudo-attribute pattern reads the metadata column its source's scan fills (the hidden variable the parser named, #594), and :db/valid-at joins as [e _ _]; plain :find variables, aggregates with :with, and window functions, over an aggregate's groups too), scan.rs (one pattern over committed key ranges via CommittedReader::key_source/live_keys, AVET ranges for cross-kind constants, then the pending range by whole runs; one entity per probe for a join), join.rs (IndexNestedLoopJoin when the pattern's entity is bound, or, under a constant attribute and entity not constant, its value (#721: per outer row the AVET ranges of the values that join-equal it and the attribute's pending facts with its key, read once per query in IdSpace::probes; an entity cell needs the keyword naming it, known only where the query made the entity from a keyword (rule heads, magic seeds), else the join becomes the planner's HashJoin); HashJoin otherwise, and on a relation; JoinKey: floats by IEEE ==, a keyword equals a ref to the entity it names), filter.rs (Filter/Bind and pseudo-attribute steps over batches; a Bind on a bound slot keeps the rows where the result is join-equal, #596), collect.rs (CollectSet, #316: [(collect-set ?e :a) ?s] per row probes [e :a ?v] as an IndexNestedLoopJoin does and binds the values as one set, #{} when none, metered; Expr::CollectSet, parsed only as a whole binding clause), anti.rs (AntiJoin, #546: a body with the same rows for every outer row runs once into a hash set of its imported-variable keys; any other body, or one whose first pattern's entity is imported, runs per outer row from seed.rs's one-row Seed, via Operator::rewind; never deduplicates), aggregate.rs (Aggregate, #547: a hash table of groups keyed on cells as Value's Ord compares values, one entry when ungrouped; count/count-distinct on cells, other aggregates on values via datalog::aggregate::Agg; groups sorted by value and finished in that order; a float group shows its least value by value_order; #641/#781 certify a complete grouping tuple as Grouping::Ordered: count/count-distinct/sum/avg, fixed-scalar extrema and Integer/nil sum-distinct; one varying ordered slot plus permitted scalar constants/repetitions, including :with (which retains Sort), no window/callback/CollectSet; one active group, bounded output batches, shared finish_group, internal DICT-cache eviction; unproved arguments and registered finalizers keep full-drain validation), order.rs and order/arguments.rs (#641/#781: snapshot-bound grouping/order and argument-domain proofs of the complete result order; safe committed integer/float/Boolean ranges can set the private ScanSpec.index hint to AVET and omit Sort, with plan.find restored from Sort.cols; entity-and-attribute scans retain EAVT; pending/staged records in that range, callbacks and CollectSet keep the original strategies; DICT comparators prove only window partition adjacency, never final Sort or aggregate group order; mixed integer/float keys can prove grouping order but keep Sort, and unique group keys discharge tie suffixes; minigraf::__input_order forces fallback under test-hooks), sort.rs (Sort, #306: the :find columns as values, in the full order of :order-by — its keys, then every other :find column ascending, by datalog::aggregate::row_order/value_order; under a :limit a heap of the best offset + limit rows; reads its whole input, dropping the DICT lookups after each batch, metered; its rows turned back into cells only as returned), limit.rs (Limit, #306: skips offset rows, passes on at most limit, then closes its child so nothing below reads more; planned after the last step, after Builder::finish, and after Sort), window.rs (Window, #547: holds each input row's passed-on cells and its named variables' values, then computes every window with datalog::aggregate::windows, the executor's code, and returns the rows in input order; #641 resolves compatible built-in windows to WindowPartitioned, retaining one partition and charged lookahead, coalescing completed partitions into bounded output batches, evicting DICT caches at internal input/result boundaries; full-input mode keeps query-lifetime caches), union.rs/distinct.rs (Union and Distinct, #546: each or branch planned as a not body is, hashed into a table or run per outer row; then one row per distinct binding, as the executor's Values compare; :distinct adds DistinctFind after aggregates/windows and before sorting/pagination, keyed on final :find columns by aggregate-group equality, with canonical float zero/NaN output including set elements, #493), cell.rs/batch.rs (id cells in columnar batches of 1,024), ids.rs (DICT and provisional ids, keyword_eid), project.rs (ids → values for returned rows only; bounded stack UUID hints share DICT paths, with cache insertion and charges in projection order, #530), relation.rs/recursive.rs (#548: rule queries; a relation per evaluated predicate holds its derived rows, a set by encode_value identity, appended per round so Old/Delta/Full are ranges; Recursive runs each stratum's semi-naive rounds, one run per same-stratum top-level invocation, a rule reading its stratum otherwise run in full every round; scans read derived rows after committed and pending ones, with nil metadata), meter.rs (#549: one Meter per query, shared by every holder of state through a Held — hash join build sides, not/or tables, distinct sets, aggregate groups, window rows, rule relations and their temporaries, IdSpace, the set a CollectSet builds; each entry charged by a per-entry estimate including retained set elements and strings before it is inserted, containers grown with try_reserve, released on rewind/close/drop; over OpenOptions::query_memory_limit, or a refused allocation, is QRY-012; Stream::close drops the operator tree, keeping its counters). mod.rs: Operator (next_batch/rewind/close/counters), Stream (the operator tree over an owned Snapshot, pulled one batch at a time). DatalogExecutor::open / execute_query run every query on it: a cursor gets the Stream (Answer::Stream), execute drains it.

  1. src/temporal.rs — UTC-only timestamp parsing (avoids chrono CVE GHSA-wcg3-cvx6-7396)

  2. src/repl.rs — Interactive REPL; TTY-aware (suppresses prompts/banner for piped input)

  3. src/db.rs — Public API: Minigraf::open/execute/query/fact_log/prepare/begin_write/checkpoint/save/verify/rebuild_indexes, WriteTransaction, IntegrityReport, OpenOptions::page_cache_size, OpenOptions::read_only (#429: shared lock, nothing written, writes are API-014), OpenOptions::query_memory_limit (#549: QRY-012, off by default), OpenOptions::transaction_chunk_facts (#550), OpenOptions::max_iterations (#861: rounds per stratum, default 1,000,000), OpenOptions::allow_overrides (#869: which query keywords may override settings, API-025; see docs/CONFIGURATION.md)

    • src/overrides.rs: Overridable (DEFAULTS, keyword; QueryMemoryLimit not in DEFAULTS, #870; MaxIterations in it, #861) and check_overrides, called after parsing at every query entry point (BrowserDb uses DEFAULTS)
    • src/cursor.rs: Cursor / Batch — owned, Send cursor returned by Minigraf::query and PreparedQuery::query (#432); pulls engine queries one internal batch at a time (close/drop stops the work; a read error after some rows comes on the next call), other queries computed at open; holds a Snapshot (one state of the store, its committed generation pinned, #543) until dropped
    • src/fact_log.rs: FactLog / FactFilter / FactRecord / FactOrder — every fact version streamed by Minigraf::fact_log (#430): tx order (multi-pass, bounded window) or storage order, key-level filters; an open log pins the committed generation (storage::pins), so checkpoints run but do not reuse its pages (API-013 deprecated, #543)
    • src/log_writer.rs: LogWriter — builds a new file from FactRecords with their tx and valid-time kept (#431): tx-ordered appends with holes (API-015/API-016), batched copy-on-write checkpoints at tx boundaries, no WAL, built at <path>.partial and renamed by finish (STG-043 if the target exists)

  4. src/wal.rs — Fact-level sidecar WAL, CRC32-protected entries, crash recovery. Like storage::verify and storage::dir_sync, not compiled for wasm32 (no file backend there)

  5. src/error.rs — Structured error codes: MinigrafError, ErrorCategory, ErrorCode registry (PRS/QRY/STG/WAL/API/INT codes); must match docs/ERROR_REFERENCE.md. MinigrafError::invalid_argument (API-017) and MinigrafError::closed (API-018) exist for language bindings

  6. src/browser/ — Browser WASM backend (browser feature): buffer.rs (BrowserBufferBackend, in-memory pages with dirty tracking; free pages a pinned generation may read are withheld from release, pins::split_released), indexeddb.rs (IndexedDB persistence); BrowserDb.query returns a BrowserCursor (JSON row batches); BrowserDb.prepare returns a BrowserPreparedQuery (execute/query with a bindings object, read by binds.rs: plain JS values or {type, value} forms, #181; a set as {type: 'set', value: [..]} or a JS Set, #316); udfs.rs: registerPredicate/registerAggregate (#180; values as {type, value} objects, a throw is QRY-014; JS values held in a thread-local table, the registry keeps ids). Tests here must be #[wasm_bindgen_test] (the module only builds for wasm32); .github/workflows/wasm.yml runs them in headless browsers, runs the WASI tests under Wasmtime, and lints both wasm targets. New file-backed integration test files need #![cfg(not(target_arch = "wasm32"))]

Data Model

struct Fact {
    entity: EntityId,  // Uuid
    attribute: String, // e.g. ":person/name"
    value: Value,
    tx_id: TxId,       // Unix ms timestamp
    tx_count: u64,     // Monotonic counter — used by :as-of N
    valid_from: i64,   // Unix ms; defaults to tx_id
    valid_to: i64,     // Unix ms; i64::MAX = forever (VALID_TIME_FOREVER)
    asserted: bool,
}

#[non_exhaustive]
enum Value { String(String), Integer(i64), Float(f64), Boolean(bool),
             Ref(Uuid), Keyword(String), Null, Set(ValueSet) }
// ValueSet: a BTreeSet of scalars (never a set), deduplicated by Value's Ord
// (-0.0 = 0.0 keeping -0.0, one NaN, keyword != ref); literal #{...} (#316)

Important: tx_count (sequential 1, 2, 3…) is what :as-of N compares against. The REPL displays tx_id (Unix ms). A single (transact [...]) command increments tx_count once regardless of how many facts it contains (transact_batch).

File Format (v8)

Page 0, 1: Meta pages A/B (alternating commits: odd generations in page 0, even in 1).
           Magic "MGRF"/"META", generation, page_count, five tree roots (EAVT, AEVT,
           AVET, VAET, DICT), free-list head, next_eid/next_iid, required_features;
           CRC over the whole page. The only commit point.
Page 2+:   Any mix of B+tree leaf/internal pages (0x61/0x62), value pages (0x51) and
           free-list pages (0x81). Every one has a 24-byte header (type, count, CRC32,
           page id, generation), verified on load.
Sidecar:   <db>.wal — v2 header records the base generation; CRC32-protected entries;
           replayed on open; deleted on checkpoint

Covering indexes: every index entry is a whole fact as a byte-comparable key (e a v tx↓ vf vt op in EAVT order), with entities and idents as sequential ids from the DICT tree, assigned at checkpoint. Strings over 64 bytes live once in value pages. Committed scans return facts in id order. A checkpoint is copy-on-write: it rewrites only touched leaves and their paths (btree::cow_insert), takes pages from the free list lazily and pushes freed ones (PageAllocator::from_chain / finish_free_list), so its cost follows the change. It writes only pages the active meta does not reference, syncs, then writes the other meta page and syncs, so a crash at any point keeps the previous checkpoint. Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are rejected (STG-028). Design: docs/superpowers/specs/2026-10-05-v8-storage-format-design.md.

Test Coverage

2104 tests (2083 active, 21 ignored; unit + integration + doc). Engine answers once checked by the executor are frozen in tests/answers/ (#551). See docs/TEST_COVERAGE.md for the full per-file breakdown.

The #760 deterministic storage suite adds three native unit tests in a separate --cfg minigraf_shuttle build, outside the default count. Run only graph::storage::deterministic:: with that cfg; ordinary unit tests do not enter its scheduler. deterministic-schedules.yml runs relevant v3 PRs on Linux/macOS/Windows plus Linux MSRV, and larger nightly/manual searches with replay artifacts. The identical main workflow activates the v3 schedule after both PRs merge. See docs/testing/deterministic-schedules.md for scope, controls and findings.

Testing conventions — see the Testing Conventions section below before writing any tests.

On v3, ordinary avg skips nil/nonnumeric values and divides the exact rounded total by the numeric count, matching window avg (#777). It returns Float, or nil for an existing group with no numbers; no matching bindings yields no rows. Sum and extrema follow the value rules from #789; ordered execution for certified grouping tuples is implemented by #781.

Current Maintainer Context

On v3, ordinary sum/sum-distinct skip nonnumbers and min/max compare all non-nil types by the window total order (#789). Empty-query conventions and all-nil min/max group dropping are preserved; INT-041/043/044 are deprecated for these paths. #781 expands Ordered Aggregate to sum/avg, fixed-scalar extrema and Integer/nil sum-distinct, with safe effective tuples and retained Sort for :with. Ordinary avg shipped in #777.

On v3, expression = / != resolves literal keywords once per operator (#774), including nested expressions and substituted prepared slots. Dynamic values do not populate the cache. This is a performance-only change, with no v2.x backport; see docs/benchmarks/774-expression-keywords.md.

v2.0.5 released (2026-10-11) — short-file open refused with STG-045 (#506), commit after a torn WAL entry no longer lost, WAL-007 (#513); backported rule-query max_results (#528), all-checkpointed WAL removal (#457) and usize::MAX close-time save (#476); weekly soak test (#392), real coverage gates (#396). v2.0.4 (2026-10-08) — BrowserDb IndexedDB flush returns an error for an unreadable dirty page (#470); golden-file v7 corpus (#391) and SIGKILL committed-data crash test (#384). v2.0.3 (2026-10-06) — data-integrity patch under the support policy: indexed queries with >65,535 uncheckpointed facts (#445), tx counter rewound on reopen next to an already-checkpointed WAL (#447). v2.0.2 (2026-09-27) — last planned v2.x feature/fix release: query fixes (#297, #405, #407, #410), directory fsync (#389), point-query/attribute-scan/checkpoint perf mitigations (#323, #381, #315), CI policy checks (#395), support policy and tiers (#397, #399, #400). v2.0.1 (2026-09-25) fixed index rebuild on open (#370). v2.0.0 introduced kernel file locking (#317, #304), structured runtime error codes (#277), OpenOptions #[non_exhaustive], and an MSRV of Rust 1.89. See CHANGELOG.md for the full rationale and release history.

Current plan (see ROADMAP.md and the milestones):

  • In-memory ancestors (#723): eligible positive three-pattern delta runs retain an entity-first alternative with the same Old/Delta/Full ranges and separately planned head slots. Each round can select it when the committed dictionary is empty, the source relation has at most 128 rows and no stored records, and a capped AEVT count favors entity probes over building an uncached pending value map. Populated committed generations keep AVET probes. No additional write-time index.
  • v2.x (main, format v7) now gets only data-integrity and security fixes as patch releases (v2.0.3 is the first); v2.0.2 was the last planned v2.x release.
  • v3.0.0 (format v8 and data integrity, plus :limit #306, optional :distinct output #493, lag/lead #182, sliding row frames #183, query profiler #185, set membership predicates #316 and prepared queries in every binding #181, UDF registration in every binding #180; v3.0.0 milestone) is developed on the long-lived v3 branch. All v3 PRs target v3; main stays untouched. CI runs on PRs to v3 the same as on PRs to main. Do not merge main into v3 routinely: main only gets backports of fixes already on v3. Shared CI and workflow changes land on both branches as separate PRs. Just before the v3.0.0 cut, merge main into v3 once (merge commit; source conflicts resolved to v3, main's v2.0.x CHANGELOG sections brought over) so the cut is a fast-forward. At the first v3 release, the current main is copied to v2 and v3 becomes the new main.
  • Input order (#641) targets v3.0.0 in three slices: committed scalar Sort omission, one-group count/count-distinct and partition-at-a-time windows are implemented in the dependent PRs for v3. The reviewed design, fallback scope and acceptance gates are in docs/superpowers/specs/2026-10-28-641-input-order-design.md.
  • Scalar key decoder proofs (#693) target v3.0.0 before the release cut. The pure decoder refactor preserves key bytes and INT-049 behavior; Kani covers round trips and bounded malformed inputs. The approved design and CI limits are in docs/superpowers/specs/2026-10-29-693-decoder-proofs.md.
  • v3.next (the release after v3.0.0) holds new features (rule persistence #241; OPFS BrowserDb #441).
  • Known v2.x data issue: same-transaction multi-valued facts (#371). Fix ships in v3.0.0. All v2.x known issues are listed in pinned issue #421 (known-issue label); data-integrity/corruption/durability issues fixed in a v2.x patch stay open until that patch is published; issues fixed on the unreleased v3 branch close on merge (closing keywords do not fire on v3, so close them with gh issue close) and stay listed in #421 (CONTRIBUTING.md).
  • Support policy and tiers: PHILOSOPHY.md §7 and §10. v2.x gets data-integrity and security fixes for 12 months after v3.0.0. Tier 1 = Rust + Python + browser WASM (from v3.0.0); other bindings are Tier 2.

Testing Conventions

Never use {:?} debug format of Result, Fact, Value, EdnValue, or any type that may transitively contain Uuid in assert!/assert_eq! message strings.

CodeQL flags this as rust/cleartext-logging. It is a false positive in tests, but it pollutes the security scan and blocks CI.

// BAD — triggers CodeQL:
assert!(result.is_ok(), "parse failed: {:?}", result);

// GOOD — plain string message:
assert!(result.is_ok(), "parse failed");

// GOOD — use unwrap/expect instead:
result.unwrap();
result.expect("parse failed");

// GOOD — assert on count/bool only:
assert_eq!(results.len(), 3, "expected 3 results");

Applies to all #[cfg(test)] modules and all tests/*.rs files.

Important Reminders

  1. Always use git worktrees for new features/bugfixes — Never make changes directly on main. Create an isolated worktree (in .worktrees/ directory) using the using-git-worktrees skill before implementing any feature or fixing any issue.
  2. Datalog is the query language — no other query language
  3. Bi-temporal is first-class — not an afterthought
  4. Single file is sacred — never break this
  5. Simplicity over features — do less, do it perfectly
  6. Test everything — no untested code
  7. Think SQLite — would SQLite do this?
  8. Long-term vision — building for decades
  9. Keep documentation synchronized — when a release or planned-work item changes, update and cross-check ALL of: CLAUDE.md (status line, test count), ROADMAP.md, README.md, docs/TEST_COVERAGE.md, CHANGELOG.md. No doc should contradict another. Also update the docs site, project-minigraf/minigraf-docs (clone at ../minigraf-docs): content/architecture.md (module/format/model changes), content/datalog-reference.md (new syntax), content/comparison.md (feature matrix), content/use-cases.md (deployment targets). It documents every release at once: wrap text that applies only from or until a version in <!-- @since vX.Y.Z --> / <!-- @until vX.Y.Z --> … <!-- @end --> rather than rewriting shared text, and add each release to its site.toml (CHANGELOG.md and docs/ERROR_REFERENCE.md are read from this repo per tag, never copied). Open a PR there; its CI checks every link in every version and deploys main. The GitHub wiki is retired: its pages only link to the site.
  10. Tag every version bump — after the final doc-sync commit: git tag -a v<x.y.z> -m "<release> — <summary>" then git push o 2E8A rigin v<x.y.z>.

When in doubt, refer to PHILOSOPHY.md and ROADMAP.md. The goal is not to be the most feature-complete graph database. The goal is to be the one that's always there when you need it, works reliably, and never gets in your way.

Be boring. Be reliable. Be Minigraf.
0