adityamukho · Oct 23, 2026 · Oct 23, 2026 · Oct 23, 2026 · Oct 23, 2026 · Oct 23, 2026
@@ -58,8 +58,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- **WAL entries of more than 1,000,000 facts replay** (#686). A WAL entry of more than 1,000,000 facts was taken as damage: replay stopped there, and the next open cut the WAL off at that entry, so after a crash, kill or close without a checkpoint that transaction and every later one in the WAL were lost, though each had committed. The writer had no such limit. Since #550 no v3 transaction writes such an entry, but a WAL left by v2.x can hold one. Replay now takes an entry's fact count as damage only when the bytes after it cannot hold that many facts. Affected every release since v0.5.0. Also fixed in v2.0.10.
 - **A recursive rule query no longer joins a rule's new facts with every magic-set fact** (#694). On the engine (#548), each round after the first runs a recursive rule from the facts the last round derived, then joins the rest of its body. The rest kept the order the planner chose for the whole body, so the next clause could share no variable with the new facts: after the magic-set rewrite of `(reach :n0 ?to)`, the body `(magic ?from) [?from :next ?mid] (reach ?mid ?to)` joined each new `reach` fact with every `magic` fact, then probed `:next` once per pair. The rest of the body is now planned with the new facts' variables bound, each next pattern sharing a variable with those bound before it when one can. Answers are unchanged. Nightly `recursion` benchmarks (bench profile): `fanout/w10_d3` 4.94 s → 22.6 ms (it was about 94 ms on the executor before #548), `chain/depth_100` 699 ms → 41.5 ms, `chain/depth_10` 1.15 ms → 0.68 ms.
-
 - **`BrowserDb` errors carry their error code** (#673). The browser API (`@minigraf/browser`) rejected with the error's top-level message only, such as `Unknown command: nosuchcommand`, while every other API reports `[PRS-001] Unknown command: nosuchcommand`. Every `BrowserDb` rejection now goes through `MinigrafError`, so it starts with its code. Host errors that have no code yet (no IndexedDB, an import blob whose length is not a multiple of the page size) are unchanged.
 - **Prepared queries use the database's rule limit** (#549). `PreparedQuery::execute` and `query` ran a rule query with the default `max_derived_facts` of 1,000,000 whatever `OpenOptions` set; they now use the database's limit, as `execute` does, and its `query_memory_limit`.
 - **An `or` branch in a rule body knows the registered predicates** (#548). A rule with `or`/`or-join` evaluated its branches with the built-in functions only, so a predicate registered with `register_predicate` in a branch (`(or [(big? ?v)] [?x :special true])`) was unknown there. Branches now get the database's functions, as the rest of the rule body does.

@@ -130,7 +130,7 @@ cargo run < demos/demo_negation.txt
    - `dir_sync.rs`: `sync_parent_dir` — fsyncs the parent directory after creating the `.graph`/WAL or deleting the WAL (no-op off Unix)
    - `packed_pages.rs`: v7 fact pages, read only for migration
    - `persistent_facts.rs`: `PersistentFactStorage` — v8 save/load, auto-migration v7→v8
-   - `persistent_facts/large_tx.rs`: large transactions (#550): a transaction whose staged facts would pass `OpenOptions::transaction_chunk_facts` (default 65,536, clamped to 1..=1,000,000, the WAL entry limit) on a file database (`Minigraf::execute` of a larger statement, or a `WriteTransaction` statement that takes it over) becomes large (WAL guard: the WAL exists, header only, before any page; cache generation bound `g + 1`; next `tx_count` reserved, allocated at the commit point; `tx_id` fixed; the pending store written as the first chunk, so in-transaction reads see the working trees alone), writes each chunk into copy-on-write working trees (per-chunk `Encoder` over the working DICT, no `SharedDictCache`; a fact the statement wrote kept once, another window of its triple `API-011`; an earlier statement's keys of a triple it writes deleted, and the triple recorded in the statement's claims tree; replaced pages: written since the statement's base released, written before it held, `g`'s freed), and commits as a checkpoint (meta `g + 1`, WAL deleted). In a `WriteTransaction` each statement has a base (roots, counters, `freed` length, `PageAllocator::begin_base` ids): a rejected statement returns to it and the transaction stays usable; staged statements are written as one chunk when it becomes large (their windows checked at its `tx_id` first). `Minigraf::execute`'s statement error rolls back (cache bound `g`, written pages evicted, handle dirty so the next checkpoint writes a meta before deleting the WAL); an I/O failure also marks the handle failed (`STG-046`). A WAL found at open also marks the handle dirty
+   - `persistent_facts/large_tx.rs`: large transactions (#550): a transaction whose staged facts would pass `OpenOptions::transaction_chunk_facts` (default 65,536, clamped to 1..=1,000,000) on a file database (`Minigraf::execute` of a larger statement, or a `WriteTransaction` statement that takes it over) becomes large (WAL guard: the WAL exists, header only, before any page; cache generation bound `g + 1`; next `tx_count` reserved, allocated at the commit point; `tx_id` fixed; the pending store written as the first chunk, so in-transaction reads see the working trees alone), writes each chunk into copy-on-write working trees (per-chunk `Encoder` over the working DICT, no `SharedDictCache`; a fact the statement wrote kept once, another window of its triple `API-011`; an earlier statement's keys of a triple it writes deleted, and the triple recorded in the statement's claims tree; replaced pages: written since the statement's base released, written before it held, `g`'s freed), and commits as a checkpoint (meta `g + 1`, WAL deleted). In a `WriteTransaction` each statement has a base (roots, counters, `freed` length, `PageAllocator::begin_base` ids): a rejected statement returns to it and the transaction stays usable; staged statements are written as one chunk when it becomes large (their windows checked at its `tx_id` first). `Minigraf::execute`'s statement error rolls back (cache bound `g`, written pages evicted, handle dirty so the next checkpoint writes a meta before deleting the WAL); an I/O failure also marks the handle failed (`STG-046`). A WAL found at open also marks the handle dirty
 
 3. **`src/query/datalog/`** — Datalog engine
    - `parser.rs`: EDN/Datalog parser — `transact`, `retract`, `query`, `rule`, `:as-of`, `:valid-at`, `not`, `not-join`; `resolve_pseudo_sources` names each per-fact pseudo-attribute's source (nearest earlier same-entity pattern, lexical scope) and its hidden metadata variable `?#<n>.<attr>`, else PRS-081 (#594)
@@ -212,7 +212,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are
 
 ## Test Coverage
 
-**1769 tests** (1755 passing, 14 ignored; unit + integration + doc).
+**1771 tests** (1757 passing, 14 ignored; unit + integration + doc).
 See `docs/TEST_COVERAGE.md` for the full per-file breakdown.
 
 **Testing conventions** — see the Testing Conventions section below before writing any tests.

@@ -25,8 +25,8 @@ use anyhow::Result;
 /// up to twice as long with random entities, for 40–55 MiB less at 1M facts.
 pub(crate) const DEFAULT_TX_CHUNK_FACTS: usize = 65_536;
 
-/// The largest chunk: a small transaction is one WAL entry, and replay
-/// refuses an entry of more facts than this (`wal.rs`, #686).
+/// The largest chunk, which bounds the facts a small transaction holds in
+/// memory and writes as one WAL entry.
 pub(crate) const MAX_TX_CHUNK_FACTS: usize = 1_000_000;
 
 #[cfg(any(test, feature = "test-oracle"))]

@@ -1093,7 +1093,7 @@ mod tests {
     }
 
     /// `OpenOptions::transaction_chunk_facts` sets the chunk size, kept
-    /// between 1 and the most facts a WAL entry holds.
+    /// between 1 and 1,000,000.
     #[test]
     fn the_chunk_size_is_an_open_option() {
         use crate::query::datalog::parser::statement::{
@@ -1103,7 +1103,6 @@ mod tests {
         assert_eq!(DEFAULT_TX_CHUNK_FACTS, 65_536);
         assert_eq!(tx_chunk_facts(0), 1);
         assert_eq!(tx_chunk_facts(usize::MAX), MAX_TX_CHUNK_FACTS);
-        assert_eq!(MAX_TX_CHUNK_FACTS, crate::wal::MAX_FACTS_PER_ENTRY);
         let dir = tempfile::tempdir().unwrap();
         for (chunk, large) in [(2, true), (3, false), (0, true)] {
             let path = dir.path().join(format!("t{chunk}.graph"));

@@ -37,11 +37,6 @@ use std::fs::{File, OpenOptions};
 use std::io::{self, Read, Seek, SeekFrom, Write};
 use std::path::Path;
 
-/// The most facts replay reads from one entry; a larger count is taken as
-/// damage. A small transaction is one entry of at most
-/// `OpenOptions::transaction_chunk_facts` facts, which is kept below this.
-pub(crate) const MAX_FACTS_PER_ENTRY: usize = 1_000_000;
-
 const WAL_MAGIC: [u8; 4] = *b"MWAL";
 const WAL_VERSION: u32 = 2;
 /// Format v7 (Minigraf v2.x) WAL version, read as base generation 1.
@@ -426,6 +421,8 @@ impl WalReader {
 /// or corrupt one or at EOF. Also returns the offset where the valid entries
 /// end: the end of the header if there are none.
 fn scan_entries(file: &mut File) -> Result<(Vec<WalEntry>, u64)> {
+    // Without the length, a corrupt count still ends at EOF, only later.
+    let file_len = file.metadata().map_or(u64::MAX, |m| m.len());
     file.seek(SeekFrom::Start(WAL_HEADER_SIZE as u64))?;
     let mut entries = Vec::new();
     let mut valid_end = WAL_HEADER_SIZE as u64;
@@ -452,13 +449,15 @@ fn scan_entries(file: &mut File) -> Result<(Vec<WalEntry>, u64)> {
         if file.read_exact(&mut num_facts_buf).is_err() {
             break; // truncated
         }
-        let num_facts = usize::try_from(u64::from_le_bytes(num_facts_buf))
-            .map_err(|_| err_coded!(ErrorCode::Wal005))?;
-
-        // Sanity cap: no legitimate entry has more than 1M facts
-        if num_facts > MAX_FACTS_PER_ENTRY {
+        // Every fact takes at least its 4-byte length, so a count the bytes
+        // left cannot hold is damage. Entries have no count limit (#686).
+        let num_facts = u64::from_le_bytes(num_facts_buf);
+        // The entry starts at `valid_end`; its facts after 20 header bytes.
+        let left = file_len.saturating_sub(valid_end.saturating_add(20));
+        if num_facts > left / 4 {
             break; // treat as corrupt entry
         }
+        let num_facts = usize::try_from(num_facts).map_err(|_| err_coded!(ErrorCode::Wal005))?;
 
         // Maximum fact size to prevent memory exhaustion from large facts
         const MAX_FACT_SIZE: usize = 10 * 1024 * 1024; // 10MB
@@ -953,22 +952,84 @@ mod tests {
         assert!(reader.read_entries().unwrap().is_empty());
     }
 
+    /// A `num_facts` too large for the bytes after it (each fact has at least
+    /// its 4-byte length) is damage: replay ends at that entry.
     #[test]
-    fn test_wal_entry_with_too_many_facts_ends_replay() {
+    fn test_wal_entry_with_more_facts_than_bytes_left_ends_replay() {
         let dir = tempfile::tempdir().unwrap();
         let path = dir.path().join("test.wal");
-        wal_with_entries(&path, 2);
+        let bytes = wal_with_entries(&path, 2);
         let one_len = {
             let p = dir.path().join("one.wal");
             wal_with_entries(&p, 1).len()
         };
         // Entry 2's num_facts, after its checksum and tx_count.
+        let count_at = one_len as u64 + 12;
+        let left = bytes.len() as u64 - (count_at + 8);
+        for num_facts in [left / 4 + 1, u64::MAX] {
+            let mut file = std::fs::OpenOptions::new().write(true).open(&path).unwrap();
+            file.seek(std::io::SeekFrom::Start(count_at)).unwrap();
+            file.write_all(&num_facts.to_le_bytes()).unwrap();
+            drop(file);
+            assert_eq!(tx_counts(&path), [1]);
+        }
+
+        // Reopening the writer cuts the WAL off at that entry.
+        drop(WalWriter::open_or_create(&path, SyncMode::Full).unwrap());
+        assert_eq!(std::fs::metadata(&path).unwrap().len(), one_len as u64);
+        assert_eq!(tx_counts(&path), [1]);
+    }
+
+    /// A count the bytes after it could hold, whose facts are cut short, ends
+    /// replay at that entry: at a fact's length or inside its bytes.
+    #[test]
+    fn test_wal_entry_with_facts_cut_short_ends_replay() {
+        let dir = tempfile::tempdir().unwrap();
+        let path = dir.path().join("test.wal");
+        let one_len = {
+            let p = dir.path().join("one.wal");
+            wal_with_entries(&p, 1).len() as u64
+        };
+
+        // Entry 2 claims a second fact; the file ends where its length would be.
+        let bytes = wal_with_entries(&path, 2);
         let mut file = std::fs::OpenOptions::new().write(true).open(&path).unwrap();
-        file.seek(std::io::SeekFrom::Start(one_len as u64 + 12))
-            .unwrap();
-        file.write_all(&1_000_001u64.to_le_bytes()).unwrap();
+        file.seek(std::io::SeekFrom::Start(one_len + 12)).unwrap();
+        file.write_all(&2u64.to_le_bytes()).unwrap();
         drop(file);
         assert_eq!(tx_counts(&path), [1]);
+
+        // Entry 2's only fact loses its last bytes.
+        std::fs::write(&path, &bytes[..bytes.len() - 3]).unwrap();
+        assert_eq!(tx_counts(&path), [1]);
+    }
+
+    /// An entry of more than 1,000,000 facts replays, and so do the entries
+    /// after it; the writer's tail cut at open keeps them (#686).
+    #[test]
+    fn test_wal_entry_of_more_than_a_million_facts_replays() {
+        const N: usize = 1_000_001;
+        let dir = tempfile::tempdir().unwrap();
+        let path = dir.path().join("test.wal");
+        let mut writer = WalWriter::open_or_create(&path, SyncMode::Full).unwrap();
+        let fact = make_fact(Uuid::new_v4(), ":n", Value::Integer(1), 1);
+        writer.append_entry(1, &vec![fact; N]).unwrap();
+        let fact = make_fact(Uuid::new_v4(), ":n", Value::Integer(2), 2);
+        writer.append_entry(2, &[fact]).unwrap();
+        drop(writer);
+        let len = std::fs::metadata(&path).unwrap().len();
+
+        // Reopening cuts off nothing, and an append lands after both.
+        let mut writer = WalWriter::open_or_create(&path, SyncMode::Full).unwrap();
+        assert_eq!(std::fs::metadata(&path).unwrap().len(), len);
+        let fact = make_fact(Uuid::new_v4(), ":n", Value::Integer(3), 3);
+        writer.append_entry(3, &[fact]).unwrap();
+        drop(writer);
+
+        let entries = WalReader::open(&path).unwrap().read_entries().unwrap();
+        let counts: Vec<u64> = entries.iter().map(|e| e.tx_count).collect();
+        assert_eq!(counts, [1, 2, 3]);
+        assert_eq!(entries[0].facts.len(), N);
     }
 
     #[test]