8000 fix!: file format v8 — keep same-transaction multi-valued facts (#371, #287) by adityamukho · Pull Request #377 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
2e21b9b
docs: design spec for index key value collision (#371, #287)
adityamukho Sep 21, 2026
c1e45ab
docs: revise #371 spec after review — legacy decoders, forced rebuild…
adityamukho Sep 24, 2026
008524d
docs: #371 spec — support only v7 as legacy format, drop v1–v6
adityamukho Sep 24, 2026
dbf27ab
docs: implementation plan for #371/#287 (file format v8)
adityamukho Sep 24, 2026
4bd6997
test: add v7 fixture with same-transaction multi-valued facts (#371, …
adityamukho Sep 24, 2026
c529896
feat!: drop support for file formats v1–v6 (#287)
adityamukho Sep 24, 2026
c74f89d
feat!: file format v8 — value bytes and asserted flag in index keys (…
adityamukho Sep 24, 2026
e425eb5
fix: v7->v8 rebuild must not rewind tx_counter or skip the index sync…
adityamukho Sep 24, 2026
8a4a890
fix: keep same-transaction multi-valued facts through query-time dedu…
adityamukho Sep 24, 2026
7f77e4f
test: add retract-collision regression tests that fail on the pre-fix…
adityamukho Sep 24, 2026
8931e84
test: v7 multi-value fixture migration and near-max-size values (#371…
adityamukho Sep 24, 2026
9d96007
docs: file format v8, v1–v6 dropped, multi-value fix (#371, #287)
adityamukho Sep 24, 2026
89b7325
fix: add valid_from/valid_to to selective_fact_fetch dedup key (#371,…
adityamukho Sep 25, 2026
ca646dc
fix: fuzz btree_page/fact_page targets against real v8 headers (#371,…
adityamukho Sep 25, 2026
edf3432
fix: stop generate_compat_fixture from overwriting the frozen v7 fixt…
adityamukho Sep 25, 2026
1be1d15
test: assert rejected future-version file is left unmodified (#371, #…
adityamukho Sep 25, 2026
5b8d83b
test: add exact-boundary max-value-size case to multi_value_test (#37…
adityamukho Sep 25, 2026
b950aa0
docs: fix stale fact_page_format doc comment (#371, #287)
adityamukho Sep 25, 2026
3bc3e45
fix: extract FactDedupKey type alias to satisfy clippy::type_complexi…
adityamukho Sep 25, 2026
bd87b95
docs: update test counts after final-review fixes (#371, #287)
adityamukho Sep 25, 2026
76cb376
fix: keep STG-003/STG-004 registered as deprecated instead of removin…
adityamukho Sep 25, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,18 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Breaking changes

- **File format v8** (#287, #371). Index keys now include the encoded value and the assert/retract flag. A v7 file (written by v2.x) is re-indexed and upgraded to v8 the first time it is opened; the upgrade is one-way and v2.x cannot open a v8 file (`STG-006`). Bindings users get this upgrade silently on open.
- **Formats v1–v6 are no longer readable.** Opening one fails with the new `STG-028`; open it once with Minigraf v2.x to upgrade it to v7 first. `STG-003` and `STG-004` are deprecated (no longer emitted, still registered, never reused). `STG-005` and `STG-006` message text changed.

### Fixed

- Two values of the same attribute written in one transaction are both returned by every query path. Previously only one came back, and entity-bound and attribute-bound queries could return different ones (#371, #287).
- A retract that removes several values of one attribute in one call now hides all of them (#371).

## v2.0.0 — 2026-08-26

This was originally slated as v1.3.0, matching the GitHub milestone name under
Expand Down
11 changes: 5 additions & 6 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,15 +104,14 @@ cargo run < demos/demo_negation.txt
- `storage.rs`: `FactStorage` — in-memory store, `transact_batch`, `retract`, `get_facts_as_of`, `get_facts_valid_at`, `net_asserted_facts`

2. **`src/storage/`** — Persistence layer
- `mod.rs`: `StorageBackend` trait, `FileHeader` v7 (84 bytes), `CommittedFactReader` / `CommittedIndexReader` traits
- `mod.rs`: `StorageBackend` trait, `FileHeader` v8 (84 bytes), `CommittedFactReader` / `CommittedIndexReader` traits
- `backend/file.rs`: Single `.graph` file backend (4KB pages, cross-platform)
- `backend/memory.rs`: In-memory backend for testing
- `index.rs`: EAVT / AEVT / AVET / VAET key types, `FactRef`, `encode_value`
- `btree_v6.rs`: On-disk B+tree (`build_btree`, `OnDiskIndexReader`, `MutexStorageBackend`)
- `btree.rs`: Legacy v5 B+tree (migration only)
- `cache.rs`: LRU page cache (`PageCache`, default 256 pages)
- `packed_pages.rs`: Packed fact pages (~25 facts/4KB page), `MAX_FACT_BYTES`
- `persistent_facts.rs`: `PersistentFactStorage` — v7 save/load, auto-migration v1–v6→v7
- `persistent_facts.rs`: `PersistentFactStorage` — v8 save/load, auto-migration v7→v8

3. **`src/query/datalog/`** — Datalog engine
- `parser.rs`: EDN/Datalog parser — `transact`, `retract`, `query`, `rule`, `:as-of`, `:valid-at`, `not`, `not-join`
Expand Down Expand Up @@ -153,7 +152,7 @@ enum Value { String(String), Integer(i64), Float(f64), Boolean(bool),

**Important**: `tx_count` (sequential 1, 2, 3…) is what `:as-of N` compares against. The REPL displays `tx_id` (Unix ms). A single `(transact [...])` command increments `tx_count` once regardless of how many facts it contains (`transact_batch`).

### File Format (v7)
### File Format (v8)

```
Page 0: Header (84 bytes) — magic "MGRF", version, page/fact counts,
Expand All @@ -163,11 +162,11 @@ After: On-disk B+tree index pages (one node per 4KB page)
Sidecar: <db>.wal — CRC32-protected WAL entries; replayed on open; deleted on checkpoint
```

Auto-migrates v1/v2/v3/v4/v5/v6 → v7 on open/checkpoint.
Auto-migrates v7 → v8 on open (index rebuild). v1–v6 are rejected (STG-028).

## Test Coverage

**1153 tests passing** (1145 passing, 8 ignored; unit + integration + doc).
**1157 tests passing** (1149 passing, 8 ignored; unit + integration + doc).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit

```bash
cargo run # interactive Datalog REPL
cargo test # run 1153 tests
cargo test # run 1157 tests
cargo run < demos/demo_recursive.txt # recursive rules demo
```

Expand Down
4 changes: 2 additions & 2 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,13 @@ Completed releases and their implementation details live in the [CHANGELOG](CHAN

## v3.0.0 — File Format Policy

**v3.0.0 supports file format v8 only.** Fixing #287's `EavtKey`/`AevtKey` missing-value-bytes data-loss bug bumps the format from v7 to v8. Support for v1–v6 is dropped at the same time. The v1–v6 → v7 auto-migration code in `persistent_facts.rs` can be removed when cutting this release; a v7→v8 migration replaces it. Any database opened at least once under a v1.x release will already be on v7; there are no known users on older formats.
**#287/#371 have landed in-tree (unreleased): file format v8, replacing v7.** Fixing #287's `EavtKey`/`AevtKey` missing-value-bytes data-loss bug bumped the format from v7 to v8; index keys now include the encoded value and the assert/retract flag. Support for v1–v6 is dropped at the same time — opening one fails with `STG-028`. A v7 file is auto-migrated (re-indexed) to v8 the first time it is opened. v3.0.0 itself has not been released yet; this work will ship as part of it. Any database opened at least once under a v1.x release will already be on v7; there are no known users on older formats.

This was the GitHub milestone named “2.0” before v2.0.0 used that version number for the kernel-locking and structured-error-code breaking changes; it was consequently renumbered to v3.0.0.

Scope:

- `EavtKey`/`AevtKey` missing-value-bytes fix and file-format v7→v8 migration (#287)
- ✅ `EavtKey`/`AevtKey` value-bytes fix, format v8, v7→v8 migration, v1–v6 dropped (#287, #371)
- `lag`/`lead` window functions (#182)
- Sliding row frames — `:rows N preceding` (#183; builds on #182)
- `PreparedQuery` over UniFFI (#181)
Expand Down
34 changes: 25 additions & 9 deletions docs/ERROR_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,9 +111,9 @@ with no `CodedError` anywhere in its chain.
| QRY-009 | Rules lock poisoned | Query Execution |
| STG-001 | Invalid header: too short | Storage |
| STG-002 | Invalid magic number: not a .graph file | Storage |
| STG-003 | Invalid v4/v5/v6 header too short | Storage |
| STG-004 | Invalid v6 header too short | Storage |
| STG-005 | Invalid v7 header too short | Storage |
| STG-003 | Invalid v4/v5/v6 header too short (deprecated) | Storage |
| STG-004 | Invalid v6 header too short (deprecated) | Storage |
| STG-005 | Invalid header too short | Storage |
| STG-006 | Unsupported format version | Storage |
| STG-007 | page_count must be greater than 0 | Storage |
| STG-008 | eavt_root_page must be less than page_count | Storage |
Expand All @@ -136,6 +136,7 @@ with no `CodedError` anywhere in its chain.
| STG-025 | Database already open in this process | Storage |
| STG-026 | Database locked by another process | Storage |
| STG-027 | Filesystem does not support file locking | Storage |
| STG-028 | Format version no longer supported | Storage |
| WAL-001 | Invalid WAL magic number | WAL |
| WAL-002 | Unsupported WAL version | WAL |
| WAL-003 | Fact serialised size exceeds maximum | WAL |
Expand Down Expand Up @@ -208,6 +209,8 @@ with no `CodedError` anywhere in its chain.
| INT-054 | Unstratifiable negative recursion cycle | Internal |
| INT-055 | Rule predicate disappeared during rollback | Internal |

Deprecated codes: STG-003 and STG-004 (pre-v7 header sizes) are deprecated as of v3.0.0 and are no longer emitted, because formats v1–v6 are rejected with STG-028 before any header-size check. They stay registered and documented, and error codes are never removed or reused.

---

## PRS — Parser Errors
Expand Down Expand Up @@ -1698,6 +1701,8 @@ See the [file format section in README](../README.md#file-format) for version hi

### STG-003 Invalid v4/v5/v6 header too short

> **Deprecated in v3.0.0.** No longer emitted: files in formats v1–v6 are rejected with [STG-028](#stg-028-format-version-no-longer-supported) before any header-size check. Kept for reference; this code will not be reused.

**Error text**: `Invalid v4/v5/v6 header: expected at least 72 bytes, got {}`

**Cause**: A file identified as format version 4, 5, or 6 is too short to hold a valid header of that version. The file is truncated at the header.
Expand All @@ -1709,6 +1714,8 @@ See the [file format section in README](../README.md#file-format) for version hi

### STG-004 Invalid v6 header too short

> **Deprecated in v3.0.0.** No longer emitted: files in formats v1–v6 are rejected with [STG-028](#stg-028-format-version-no-longer-supported) before any header-size check. Kept for reference; this code will not be reused.

**Error text**: `Invalid v6 header: expected 80 bytes, got {}`

**Cause**: A file identified as format version 6 is shorter than the required 80-byte v6 header. The file is truncated.
Expand All @@ -1718,20 +1725,18 @@ See the [file format section in README](../README.md#file-format) for version hi

**Scenario**: A v6-format `.graph` file was written by an older pre-release version and its header is incomplete.

### STG-005 Invalid v7 header too short
### STG-005 Invalid header too short

**Error text**: `Invalid v7 header: expected 84 bytes, got {}`
**Error text**: `Invalid header: expected 84 bytes, got {}`

**Cause**: A file identified as format version 7 (current format) is shorter than the required 84-byte header. The file is truncated.
**Cause**: The file declares a supported format version (7 or later) but is shorter than the 84-byte header. The file is truncated.

**Resolution**:
- Restore from backup. If the file is newly created, delete it. See the [file format section in README](../README.md#file-format).

**Scenario**: A v7-format `.graph` file was partially written by a crash during header initialisation.

### STG-006 Unsupported format version

**Error text**: `Unsupported format version: {} (supported: 1-{})`
**Error text**: `Unsupported format version: {} (supported: 7-{})`

**Cause**: The file was written by a newer version of Minigraf than is currently installed. The format version number in the header is outside the range this library can read.

Expand Down Expand Up @@ -1984,6 +1989,17 @@ See the [file format section in README](../README.md#file-format) for version hi

**NFSv3 `nolock` is not covered by this error — it is a silent, undetectable gap (#334).** An export explicitly mounted with `-o nolock` behaves differently from plain "no `lockd`": the Linux NFS client serves `flock`/OFD locks out of its own local, in-kernel lock table instead of going over NLM to the server. `try_lock` sees an ordinary local success, `classify` takes the same branch it would on a filesystem where locking genuinely works, and this error is never raised — `allow_unlocked` is never consulted because the code never learns the mount can't really lock. Two separate client hosts writing the same `nolock` export each get a false `Ok(())` from their own kernel with zero cross-host coordination, and can silently corrupt the file. There is no way to detect this from inside `try_lock`'s result, so `nolock` NFSv3 exports are an unsupported deployment for multi-writer use, the same way running mixed Minigraf versions against one file is unsupported (see the kernel-locking CHANGELOG entry) — avoid `nolock` exports rather than relying on this check to catch them.

### STG-028 Format version no longer supported

**Error text**: `Format version {} is no longer supported (oldest supported: {}). Open the file once with Minigraf v2.x to upgrade it to format v7, then open it with this version.`

**Cause**: The file was last written by Minigraf v1.x or earlier in a format older than v7. Minigraf v3.0.0 and later read only format v7 (written by v2.x) and v8.

**Resolution**:
- Open the file once with Minigraf v2.x, which upgrades it to format v7, then open it with this version. The file is not modified by the failed open.

**Scenario**: A `.graph` file created with Minigraf v1.x and never opened with v2.x.

---

## WAL — Write-Ahead Log Errors
Expand Down
15 changes: 10 additions & 5 deletions docs/TEST_COVERAGE.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,22 @@
# Minigraf Test Coverage

## v2.0.0 Coverage Summary
## Coverage Summary

**Verified**: 2026-08-26 with `cargo test --quiet`
**Verified**: 2026-09-25 with `cargo test --quiet`

**Result**: 1,145 passing tests and 8 ignored tests (1,153 total)
**Result**: 1,149 passing tests and 8 ignored tests (1,157 total)

The v2.0.0 suite covers the public database API, the Datalog engine, storage and recovery paths, and the release’s locking and structured-error-code changes.
This supersedes the v2.0.0-era count (1,145 passing, 8 ignored) with the unreleased
file-format-v8 work (#287, #371): the index-key value-collision fix, the v7→v8
migration, and the drop of formats v1–v6. It otherwise still covers the public
database API, the Datalog engine, storage and recovery paths, and v2.0.0's locking
and structured-error-code changes.

### Covered Areas

- **Core database behavior**: in-memory and file-backed operation, transactions, checkpoints, retractions, bi-temporal queries, recursive rules, prepared statements, aggregation, expressions, disjunction, window functions, UDFs, and magic-sets evaluation.
- **Storage correctness**: packed pages, B+tree indexes and range scans, cache behavior, file-header validation, migration, WAL replay, checksums, corruption handling, and fault injection.
- **Multi-value and index-key correctness** (`tests/multi_value_test.rs`, 10 tests): same-transaction multi-valued facts surviving query-time dedup, retracting several values of one attribute in one call, valid-time stints (facts differing only in valid time), maximum-value-size boundaries, and format v8 index keys (value bytes + assert/retract flag) (#287, #371).
- **Storage correctness**: packed pages, B+tree indexes and range scans, cache behavior, file-header validation, migration (including v7→v8 index rebuild and v1–v6 rejection via `STG-028`), WAL replay, checksums, corruption handling, and fault injection.
- **Reliability and concurrency**: concurrent reads and writes, rollback behavior, same-process handle exclusion, cross-process locking, PID namespaces, NFS and container lock behavior, and SIGKILL recovery.
- **Public diagnostics**: structured parser, query, storage, WAL, API, and internal error codes; the error-code registry is checked against [`ERROR_REFERENCE.md`](ERROR_REFERENCE.md).
- **Compatibility and quality**: cross-platform format compatibility, property-based tests, long-haul smoke coverage, XTDB/Datomic semantic compatibility, grammar conformance, and rustdoc examples.
Expand Down
Loading
Loading
0