adityamukho · Oct 6, 2026 · Oct 6, 2026 · Oct 6, 2026 · Oct 6, 2026 · Oct 6, 2026
@@ -11,13 +11,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - **File format v8** (#287, #371). Index keys now include the encoded value and the assert/retract flag. A v7 file (written by v2.x) is re-indexed and upgraded to v8 the first time it is opened; the upgrade is one-way and v2.x cannot open a v8 file (`STG-006`). Bindings users get this upgrade silently on open.
 - **Crash-atomic checkpoints with verified pages** (v8 storage format, PR 2 of 5; #374, #388). Two alternating meta pages replace the 84-byte file header and are the only commit point: a checkpoint writes only pages the last commit does not use, syncs, then writes the other meta page and syncs. A crash or torn write at any point leaves the previous checkpoint intact, so opening never rebuilds an index, and the whole-file index checksum is gone. Every other page carries a 24-byte header (type, CRC32, its own page id, the checkpoint generation that wrote it), checked on every read: a torn, corrupted, misplaced or stale page now fails with `STG-029`, `STG-030` or `STG-031` instead of being read. If the newest meta page is damaged after its WAL was already deleted, open fails with `STG-033` rather than silently going back to the previous checkpoint. New codes `STG-029`–`STG-035`. The WAL header gains a base generation (WAL version 2); a v2.x WAL beside a migrated file is still replayed. Pages freed by a checkpoint are reused by later ones; checkpoints still rewrite all four indexes until copy-on-write insert lands (PR 4). Files written by a v3.0.0 development build before this change fail with `STG-032`.
-- **The maximum fact size for file-backed databases is 4,052 bytes** (was 4,080), so that every index key derived from a fact fits in an index page. Index pages now reject an entry that does not fit instead of writing an overlapping page.
+- **Covering indexes with dictionary-coded keys** (v8 storage format, PR 3 of 5; #433). Every index entry now holds the whole fact as a byte-comparable key, so a query reads only index pages, never a separate fact page. Entities and idents (attribute names and keyword values) get sequential internal ids from a dictionary stored in the file, and strings longer than 64 bytes are stored once in value pages and deduplicated. Fact pages are gone. A file with 100,000 facts (10 attributes per entity) takes 88 bytes per fact, down from 314. Facts from committed data now come back in internal-id order (entity, then attribute) rather than insertion or UUID order; query results are sets, and code that relied on an order must sort. New codes `STG-036` (dictionary entry missing) and `STG-037` (one transaction with two timestamps: a checkpoint or v7 upgrade refuses to change a stored timestamp). Files written by a v3.0.0 development build before this change fail with `STG-032`.
+- **Size limits for file-backed databases:** a string value may be up to 4,068 bytes, whatever the rest of the fact, and an attribute name or keyword value up to 1,024 bytes. Both are checked when the transaction is written and fail with `WAL-003`, whose message now names the value. This replaces the 4,080-byte limit on a whole serialised fact (4,052 in earlier v3.0.0 development builds). In-memory databases have no limit.
 - **Formats v1–v6 are no longer readable.** Opening one fails with the new `STG-028`; open it once with Minigraf v2.x to upgrade it to v7 first. `STG-003` and `STG-004` are deprecated (no longer emitted, still registered, never reused). `STG-005` and `STG-006` message text changed.
 
 ### Fixed
 
 - Two values of the same attribute written in one transaction are both returned by every query path. Previously only one came back, and entity-bound and attribute-bound queries could return different ones (#371, #287).
 - A retract that removes several values of one attribute in one call now hides all of them (#371).
+- Reopening next to a WAL whose entries were all already checkpointed no longer rewinds the transaction counter, so later transactions no longer reuse `tx_count` values and `:as-of` history stays correct (#447; also fixed in v2.x).
 
 ### Internal
 

@@ -104,15 +104,21 @@ cargo run < demos/demo_negation.txt
    - `storage.rs`: `FactStorage` — in-memory store, `transact_batch`, `retract`, `get_facts_as_of`, `get_facts_valid_at`, `net_asserted_facts`, `get_facts_by_entity_attribute_indexed`
 
 2. **`src/storage/`** — Persistence layer
-   - `mod.rs`: `StorageBackend` trait, `FileHeader` v8 (84 bytes), `CommittedFactReader` / `CommittedIndexReader` traits
+   - `mod.rs`: `StorageBackend` trait, `LegacyHeaderV7` (v7 migration only), `CommittedReader` trait
    - `backend/file.rs`: Single `.graph` file backend (4KB pages, cross-platform)
    - `backend/memory.rs`: In-memory backend for testing
    - `backend/fault_inject.rs`: `FaultInjectingBackend` — injects I/O errors for durability tests (test builds only)
-   - `index.rs`: EAVT / AEVT / AVET / VAET key types, `FactRef`, `encode_value`
-   - `btree_v6.rs`: On-disk B+tree (`build_btree`, `rebuild_btree_incremental`, `OnDiskIndexReader`, `MutexStorageBackend`)
+   - `index.rs`: pending (uncheckpointed) EAVT/AEVT keys over UUIDs and strings, `encode_value`
+   - `keys.rs`: byte-comparable v8 keys (FDB integers, value tags, `tx↓`, FOREVER), DICT keys, `MAX_VALUE_BYTES`, `MAX_IDENT_BYTES`
+   - `node.rs`: prefix-compressed leaf and shortest-separator internal node codecs
+   - `btree.rs`: On-disk B+tree over byte keys (`build_btree`, `rebuild_btree_incremental`, `LeafCursor` with `seek`, `prefix_scan`, `get`, `MutexStorageBackend`)
+   - `dict.rs`: `DictReader` (id ↔ UUID/ident, tx timestamps, long values) and `Encoder` (checkpoint-time id assignment and key building)
+   - `value_pages.rs`: append-only value pages for strings over 64 bytes
+   - `reader.rs`: `OnDiskReader` — covering reads of committed facts (`CommittedReader`)
+   - `meta.rs` / `page.rs` / `freelist.rs`: meta pages A/B, the common page header and allocator, free-list chain
    - `cache.rs`: LRU page cache (`PageCache`, default 256 pages)
    - `dir_sync.rs`: `sync_parent_dir` — fsyncs the parent directory after creating the `.graph`/WAL or deleting the WAL (no-op off Unix)
-   - `packed_pages.rs`: Packed fact pages (~25 facts/4KB page), `MAX_FACT_BYTES`
+   - `packed_pages.rs`: v7 fact pages, read only for migration
    - `persistent_facts.rs`: `PersistentFactStorage` — v8 save/load, auto-migration v7→v8
 
 3. **`src/query/datalog/`** — Datalog engine
@@ -163,23 +169,28 @@ enum Value { String(String), Integer(i64), Float(f64), Boolean(bool),
 
 ```
 Page 0, 1: Meta pages A/B (alternating commits: odd generations in page 0, even in 1).
-           Magic "MGRF"/"META", generation, page_count, index roots, free-list head,
-           required_features; CRC over the whole page. The only commit point.
-Page 2+:   Any mix of fact pages (0x41, interim until covering indexes), B+tree
-           leaf/internal pages (0x61/0x62) and free-list pages (0x81). Every one has
-           a 24-byte header (type, count, CRC32, page id, generation), verified on load.
+           Magic "MGRF"/"META", generation, page_count, five tree roots (EAVT, AEVT,
+           AVET, VAET, DICT), free-list head, next_eid/next_iid, required_features;
+           CRC over the whole page. The only commit point.
+Page 2+:   Any mix of B+tree leaf/internal pages (0x61/0x62), value pages (0x51) and
+           free-list pages (0x81). Every one has a 24-byte header (type, count, CRC32,
+           page id, generation), verified on load.
 Sidecar:   <db>.wal — v2 header records the base generation; CRC32-protected entries;
            replayed on open; deleted on checkpoint
 ```
 
-A checkpoint writes only pages the active meta does not reference, syncs, then writes
-the other meta page and syncs, so a crash at any point keeps the previous checkpoint.
+Covering indexes: every index entry is a whole fact as a byte-comparable key
+(`e a v tx↓ vf vt op` in EAVT order), with entities and idents as sequential ids from
+the DICT tree, assigned at checkpoint. Strings over 64 bytes live once in value pages.
+Committed scans return facts in id order. A checkpoint writes only pages the active
+meta does not reference, syncs, then writes the other meta page and syncs, so a crash
+at any point keeps the previous checkpoint.
 Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are rejected
 (STG-028). Design: `docs/superpowers/specs/2026-10-05-v8-storage-format-design.md`.
 
 ## Test Coverage
 
-**1233 tests** (1225 passing, 8 ignored; unit + integration + doc).
+**1223 tests** (1215 passing, 8 ignored; unit + integration + doc).
 See `docs/TEST_COVERAGE.md` for the full per-file breakdown.
 
 **Testing conventions** — see the Testing Conventions section below before writing any tests.

@@ -144,9 +144,11 @@ with no `CodedError` anywhere in its chain.
 | STG-033 | Meta page damaged after commit | Storage |
 | STG-034 | Unsupported file feature | Storage |
 | STG-035 | Free-list inconsistency | Storage |
+| STG-036 | Dictionary entry missing | Storage |
+| STG-037 | Transaction with two timestamps | Storage |
 | WAL-001 | Invalid WAL magic number | WAL |
 | WAL-002 | Unsupported WAL version | WAL |
-| WAL-003 | Fact serialised size exceeds maximum | WAL |
+| WAL-003 | Value size exceeds maximum | WAL |
 | WAL-004 | Fact serialised size exceeds u32 range | WAL |
 | WAL-005 | WAL num_facts exceeds platform usize | WAL |
 | WAL-006 | Failed to delete WAL file | WAL |
@@ -2047,7 +2049,7 @@ See the [file format section in README](../README.md#file-format) for version hi
 
 **Error text**: `No valid meta page: the file is not a Minigraf v7 or v8 database, or both meta pages are damaged`
 
-**Cause**: Neither meta page (page 0 or page 1) passes its checksum, page 0 is not a valid format v7 header, and no migration backup meta page can be used. Pre-release v8 files written during v3.0.0 development are also rejected this way.
+**Cause**: Neither meta page (page 0 or page 1) passes its checksum, page 0 is not a valid format v7 header, and no migration backup meta page can be used. Pre-release v8 files written during v3.0.0 development are also rejected this way: those with a single 84-byte header, and those with meta pages but no dictionary tree (written before covering keys).
 
 **Resolution**:
 - Check that the path points to a Minigraf database.
@@ -2090,6 +2092,32 @@ See the [file format section in README](../README.md#file-format) for version hi
 
 ---
 
+### STG-036 Dictionary entry missing
+
+**Error text**: `Dictionary entry missing: {}`
+
+**Cause**: An index entry refers to an entity id, ident id, transaction or long value that has no entry in the file's dictionary (DICT tree). The file is damaged.
+
+**Resolution**:
+- Restore from backup.
+
+**Scenario**: A DICT leaf lost to a bad sector while the index trees that refer to its ids are intact.
+
+---
+
+### STG-037 Transaction with two timestamps
+
+**Error text**: `Transaction {} has two timestamps: {} and {}`
+
+**Cause**: A checkpoint or a v7 migration found facts with the same `tx_count` but different `tx_id` values. Format v8 records one timestamp per transaction, and Minigraf refuses to change a stored timestamp. Every write path stamps one `tx_id` per `tx_count`, so this means the facts were written outside those paths or the file is damaged.
+
+**Resolution**:
+- Keep using the release that wrote the file and report the issue, with the transaction number from the message.
+
+**Scenario**: A v7 file whose facts were edited by an external tool so that two transactions share one `tx_count`.
+
+---
+
 ## WAL — Write-Ahead Log Errors
 
 WAL errors relate to the sidecar `.wal` file written alongside the `.graph` file.
@@ -2120,11 +2148,11 @@ The WAL is replayed on open and deleted on checkpoint.
 
 **Scenario**: A `.wal` file written by a pre-release version of Minigraf is opened with the stable release, which uses a different WAL version number — e.g. `Unsupported WAL version: 3 (expected 2)`.
 
-### WAL-003 Fact serialised size exceeds maximum
+### WAL-003 Value size exceeds maximum
 
-**Error text**: `Fact serialised size {} bytes exceeds maximum {} bytes. Store large payloads externally and reference them with a Value::String URL/path or Value::Ref entity ID.`
+**Error text**: `Value of {} bytes exceeds the maximum of {} bytes. Store large payloads externally and reference them with a Value::String URL/path or Value::Ref entity ID.`
 
-**Cause**: A single fact's serialised size exceeds the WAL entry limit (~512 KB). This typically means a `Value::String` attribute value contains very large content such as raw document text, a base64-encoded image, or binary data.
+**Cause**: A string value is longer than `MAX_VALUE_BYTES` (4,068 bytes, one value page), or an attribute name or keyword value is longer than 1,024 bytes. This typically means a `Value::String` contains very large content such as raw document text, a base64-encoded image, or binary data. In-memory databases have no WAL and no limit.
 
 **Resolution**:
 - Store large payloads in an external file or object store.
@@ -2142,7 +2170,7 @@ The WAL is replayed on open and deleted on checkpoint.
 
 **Error text**: `fact serialised size {} exceeds u32 range`
 
-**Cause**: The serialised size of a single fact exceeds `u32::MAX` (~4 GB). This is practically unreachable — WAL-003's ~512 KB limit fires first.
+**Cause**: The serialised size of a single fact exceeds `u32::MAX` (~4 GB). This is practically unreachable: WAL-003's value limits fire first.
 
 **Resolution**:
 - This should not occur under normal operation.