8000 fix: one current valid-time window per (e, a, v) (#435) by adityamukho · Pull Request #460 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **A damaged page is reported, not worked around** (v8 storage format, PR 5 of 5). A query whose indexed lookup reaches a damaged page now fails with that page's `STG-029`/`STG-030`/`STG-031`. Before, the query fell back to a full scan, which could hide the damage. Tests damage an index leaf, an internal node, a dictionary leaf, a value page and a free-list page in turn. Every read or checkpoint that reaches the page fails, and the file still opens at its last checkpoint. Measured at 1.15M facts in #433's shape: 142 bytes per fact, history included. A checkpoint after one new fact takes 6.1 ms at 1M facts. Concurrent point queries are 19–31 % faster than before this change: reads reuse the entity and attribute the caller passed, and share cached ident and timestamp lookups.
- **Size limits for file-backed databases:** a string value may be up to 4,068 bytes, whatever the rest of the fact, and an attribute name or keyword value up to 1,024 bytes. Both are checked when the transaction is written and fail with `WAL-003`, whose message now names the value. This replaces the 4,080-byte limit on a whole serialised fact (4,052 in earlier v3.0.0 development builds). In-memory databases have no limit.
- **Formats v1–v6 are no longer readable.** Opening one fails with the new `STG-028`; open it once with Minigraf v2.x to upgrade it to v7 first. `STG-003` and `STG-004` are deprecated (no longer emitted, still registered, never reused). `STG-005` and `STG-006` message text changed.
- **One current valid-time window per fact** (#435). At any transaction time, each `(entity, attribute, value)` now has exactly one current valid-time window: the one from its latest `transact`. A later assertion of the same fact replaces the earlier window, which stays visible through `:as-of` of the earlier transaction. Before, every window ever asserted stayed live until a `retract`, so closing an open window with `(transact {:valid-from vf :valid-to D} ...)` left the open one live and `:valid-at` after `D` still found the fact. Closing, extending or reopening a window is now a plain `transact` with the new bounds; `retract` still withdraws the fact. **This changes query results for existing data:** a fact asserted in several transactions with different windows is now valid only in its latest window at the current transaction time. A fact that was true over two separate periods can no longer have both windows live at once; model each period as its own entity, or read earlier periods with `:as-of`. Asserting one fact with two different windows in one transaction (one `transact`, or several `execute` calls in one `WriteTransaction`) is rejected with the new `API-011`, and nothing is written. Data written before this check with two windows in one transaction keeps both.

### Added

Expand All @@ -30,7 +31,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Internal

- **Committed reads hide superseded history on index keys** (#379). A query's committed facts are now reduced to the ones that survive net-assert (newest assertion per valid-time window, newer than every retraction) while walking the index keys, before any fact is built. Only surviving entries are decoded and translated through the dictionary. A transaction that retracted a triple hides everything older in it, and a long run of hidden entries is skipped with one `seek`. `:as-of N` point and attribute queries now take the same index path instead of reading every fact in the file. On a checkpointed file with 2,000 superseded values, a point query takes 1.03 ms instead of 4.61 ms, an attribute scan 1.24 ms instead of 5.87 ms, and an `:as-of` point query 23 µs instead of 5.6 ms (at depth 1). Results are unchanged: the executor still applies net-assert to committed and pending facts together, and a pending fact is always newer than a committed one.
- **Committed reads hide superseded history on index keys** (#379). A query's committed facts are now reduced to the ones that survive net-assert (each fact's assertion from its newest transaction, unless that transaction retracted it; #435) while walking the index keys, before any fact is built. Only surviving entries are decoded and translated through the dictionary. A transaction that retracted a triple hides everything older in it, and a long run of hidden entries is skipped with one `seek`. `:as-of N` point and attribute queries now take the same index path instead of reading every fact in the file. On a checkpointed file with 2,000 superseded values, a point query takes 1.03 ms instead of 4.61 ms, an attribute scan 1.24 ms instead of 5.87 ms, and an `:as-of` point query 23 µs instead of 5.6 ms (at depth 1). Results are unchanged: the executor still applies net-assert to committed and pending facts together, and a pending fact is always newer than a committed one.
- **B+tree reads no longer follow leaf sibling links** (v8 storage format, PR 1 of 5; #374). Every index scan uses a cursor that walks down from the root and keeps the path, with a forward `seek` that climbs only as far as needed. This prepares for copy-on-write index pages, which cannot keep sibling links. The file format and query results are unchanged. Concurrent 100k-fact scans take 25–37 % less time and checkpointing after one fact at 10k facts 9 % less; single point lookups and attribute scans are 2–6 % slower (about 0.4 µs per lookup). A corrupted child pointer now fails with STG-013 (wrong page type) or INT-049 (cycle) instead of following a bad link.

### Tests
Expand Down
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -194,7 +194,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are

## Test Coverage

**1252 tests** (1243 passing, 9 ignored; unit + integration + doc).
**1267 tests** (1258 passing, 9 ignored; unit + integration + doc).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit

```bash
cargo run # interactive Datalog REPL
cargo test # run 1252 tests
cargo test # run 1267 tests
cargo run < demos/demo_recursive.txt # recursive rules demo
```

Expand Down
1 change: 1 addition & 0 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ Scope — format and storage integrity:
The format is frozen for the golden-file corpus (#391).
- Index `verify` and public `rebuild_indexes()` (#373; done on `v3`)
- Net-assert on v8 keys before resolving facts, for O(live) point queries (#379; done on `v3`)
- One current valid-time window per `(e, a, v)`: a later assertion replaces the earlier window, and two windows of one fact in one transaction are `API-011` (#435; done on `v3`)
- `btree_page` fuzz target that reaches node decoding (#375)

Scope — release-gate testing and documentation:
Expand Down
24 changes: 24 additions & 0 deletions docs/ERROR_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,7 @@ with no `CodedError` anywhere in its chain.
| API-008 | Function registry lock poisoned | Database API |
| API-009 | WAL not initialized | Database API |
| API-010 | Query with bind slots passed to execute() | Database API |
| API-011 | Two valid-time windows for one fact in one transaction | Database API |
| INT-000 | Unclassified internal error | Internal |
| INT-001 | WriteTransaction already in progress on this thread | Internal |
| INT-002 | Invalid entity (API layer) | Internal |
Expand Down Expand Up @@ -2428,6 +2429,29 @@ let result = pq.execute(&[

**Scenario**: A query copied from the prepared-query docs is pasted into the REPL or passed to `db.execute()` unchanged.

### API-011 Two valid-time windows for one fact in one transaction

**Error text**: `one transaction asserts the same value of {} with two valid-time windows; assert it once with its final window`

**Cause**: One transaction asserts the same entity, attribute and value twice with different `:valid-from`/`:valid-to` bounds. At any transaction time a fact has exactly one current valid-time window, and the window of the latest transaction applies. Two windows in the same transaction leave no latest one, so the transaction is rejected and nothing is written. The `{}` is the attribute. This can come from one `(transact ...)` with per-fact bounds, or from several `execute()` calls inside one `WriteTransaction`.

**Resolution**:
- Assert the fact once per transaction, with the window it should have.
- To change a window later (close it, extend it, reopen it), `transact` the same fact again with the new bounds. The earlier window stays visible through `:as-of`.
- A fact that was true over two separate periods cannot have both windows live at once. Model each period as its own entity (for example an `:employment` entity with `:employment/salary`), or keep the latest period current and read earlier ones with `:as-of`.

```clojure
;; Rejected: two windows for [:alice :salary 100000] in one transaction
(transact [[:alice :salary 100000 {:valid-from "2020-01-01" :valid-to "2022-01-01"}]
[:alice :salary 100000 {:valid-from "2024-01-01"}]])

;; Accepted: close the open window in a later transaction
(transact {:valid-from "2020-01-01"} [[:alice :salary 100000]])
(transact {:valid-from "2020-01-01" :valid-to "2022-01-01"} [[:alice :salary 100000]])
```

**Scenario**: A batch import writes the same fact once per period it was true, all in one transaction.

---

## INT — Internal Errors
Expand Down
4 changes: 2 additions & 2 deletions docs/TEST_COVERAGE.md

Large diffs are not rendered by default.

4 changes: 4 additions & 0 deletions docs/superpowers/specs/2026-10-06-live-key-scans-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,10 @@
**Milestone:** v3.0.0, branch `v3`, file format v8 (no format change)
**Builds on:** `2026-10-05-v8-storage-format-design.md` §5.3, §7

> **Superseded in part by #435** (`2026-10-06-one-window-per-triple-design.md`): net-assert now
> keeps each triple's newest transaction, not the newest assertion per window, so the walk
> stops at the first transaction group of each triple. The per-window rule below is history.

## 1. Problem

Every non-`:as-of` query fetches facts, then applies `net_asserted_facts` (keep an
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# One current valid-time window per triple — Design

**Issue:** #435, tracker #383
**Milestone:** v3.0.0, branch `v3` (no format change; changes query results)
**Builds on:** `2026-10-06-live-key-scans-design.md` (#379)

## 1. Model

For each `(e, a, v)`, transaction time N has exactly one current valid-time window: the
window of the latest assertion with `tx_count ≤ N`, provided it is newer than the
triple's last retraction. `:as-of N` selects that window; `:valid-at` is then checked
against it only. A later assertion replaces the earlier window, which stays visible
through `:as-of` of the earlier transaction. Closing, extending or reopening a window
is a `transact` with the new bounds. `retract` still withdraws the triple.

## 2. Defect

`net_asserted_facts` picked a winner per `(e, a, v, vf, vt)`, so every window ever
asserted stayed live until a retraction. `[vf, ∞)` followed by `[vf, D)` left both live.

## 3. Changes

- `net_asserted_facts` (`src/graph/storage.rs`): one map keyed by `(e, a, v)` holding
the first latest assertion's index and `tx_count`, plus a `tied` flag set when another
assertion has the same `tx_count`. Untied triples keep one record with no extra work.
Tied triples (only from data written before §4) take a second pass that keeps each
window of the latest transaction once, which also keeps the function idempotent
under duplicated input.
- On-disk walk (`OnDiskReader::live_scan`, `src/storage/reader.rs`): the first
transaction group of a triple at or before `as_of` decides it. A retraction in it
hides the triple, otherwise its assertions are emitted. Every older entry is dead and
skipped (with the existing seek after 8). The per-triple window set is gone.
- `check_one_window_per_triple`: two assertions of one triple with different
`(vf, vt)` in one transaction are rejected with **API-011**. It runs on
`Minigraf::execute` transacts and `WriteTransaction::commit` before a `tx_count` is
allocated or the WAL is written, and in `FactStorage::transact`/`transact_batch`.
Unresolved `valid_from`s all hold `VALID_FROM_USE_TX_TIME`, so repeats without bounds
compare equal, as they will after stamping. Identical repeats and retractions are
allowed. WAL replay, migration and `load_fact` are not checked: old data must open.

## 4. Data written before the check

v2.x and earlier v3 builds could store two windows of one triple in one transaction.
Both stay live (once each) until a later transaction of that triple. In-memory and
on-disk paths agree on this, and no open or migration fails because of it.

## 5. Tests

- Unit: replace, close, retraction-withdraws, same-transaction retraction, legacy tied
windows; randomized histories against a per-triple model (replaces the pre-#323
per-window oracle); API-011 accepts repeats, other values and retractions; a rejected
`transact_batch` takes no `tx_count`.
- On disk: hand-written history (closing, same-transaction retract, legacy pair) at every
`as_of`, before and after reopen; fails on the per-window walk.
- `tests/valid_time_window_test.rs`: the issue's table, close, extend/reopen,
retract-then-reassert, API-011 from `transact` and `WriteTransaction`, in memory,
checkpointed, and reopened, through EAVT and AEVT queries.
- Revised: `bitemporal_test::test_same_eav_later_valid_time_interval_replaces_earlier`,
`multi_value_test::multi_value_time_stints_visible_at_correct_valid_time_on_every_path`.
3 changes: 3 additions & 0 deletions src/browser/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -266,6 +266,9 @@ impl BrowserDb {
let (dirty_pages, result_json) = {
let mut inner = self.inner.borrow_mut();

// Before allocating: a rejected transaction takes no tx_count (#435).
crate::graph::storage::check_one_window_per_triple(&facts)
.map_err(|e| JsValue::from_str(&e.to_string()))?;
let tx_count = inner.fact_storage.allocate_tx_count();
let tx_id = tx_id_now();

Expand Down
6 changes: 6 additions & 0 deletions src/db.rs
10BC8
Original file line number Diff line number Diff line change
Expand Up @@ -626,6 +626,8 @@ impl Minigraf {
}
_ => return Err(err_coded!(ErrorCode::Api002)),
};
// Before allocating: a rejected transaction takes no tx_count.
crate::graph::storage::check_one_window_per_triple(&stamped)?;

let tx_count = self.inner.fact_storage.allocate_tx_count();
let tx_id = crate::graph::types::tx_id_now();
Expand Down Expand Up @@ -1325,6 +1327,10 @@ impl<'a> WriteTransaction<'a> {
let facts_to_commit = std::mem::take(&mut self.pending_facts);

if !facts_to_commit.is_empty() {
// Before allocating: a rejected transaction takes no tx_count.
// Unresolved `valid_from`s all equal `VALID_FROM_USE_TX_TIME`, so
// they compare as they will after stamping.
crate::graph::storage::check_one_window_per_triple(&facts_to_commit)?;
let tx_count = self.inner.fact_storage.allocate_tx_count();
let tx_id = crate::graph::types::tx_id_now();

Expand Down
9 changes: 9 additions & 0 deletions src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,7 @@ pub(crate) enum ErrorCode {
Api008,
Api009,
Api010,
Api011,
Int001,
Int002,
Int003,
Expand Down Expand Up @@ -1136,6 +1137,12 @@ pub(crate) const REGISTRY: &[(ErrorCode, &str, &str, ErrorCategory)] = &[
"query contains bind slots ({}); use prepare() and execute the prepared query with bind values",
ErrorCategory::Api,
),
(
ErrorCode::Api011,
"API-011",
"one transaction asserts the same value of {} with two valid-time windows; assert it once with its final window",
ErrorCategory::Api,
),
(
ErrorCode::Int001,
"INT-001",
Expand Down Expand Up @@ -1884,6 +1891,7 @@ mod tests {
| ErrorCode::Api008
| ErrorCode::Api009
| ErrorCode::Api010
| ErrorCode::Api011
| ErrorCode::Int001
| ErrorCode::Int002
| ErrorCode::Int003
Expand Down Expand Up @@ -2089,6 +2097,7 @@ mod tests {
ErrorCode::Api008,
ErrorCode::Api009,
ErrorCode::Api010,
ErrorCode::Api011,
ErrorCode::Int001,
ErrorCode::Int002,
ErrorCode::Int003,
Expand Down
Loading
Loading
0