8000 fix: reject empty valid-time windows; last statement wins in a transaction (#436, #477) by adityamukho · Pull Request #480 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions CHANGELOG.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -197,7 +197,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are

## Test Coverage

**1348 tests** (1339 passing, 9 ignored; unit + integration + doc).
**1360 tests** (1351 passing, 9 ignored; unit + integration + doc).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit

```bash
cargo run # interactive Datalog REPL
cargo test # run 1348 tests
cargo test # run 1360 tests
cargo run < demos/demo_recursive.txt # recursive rules demo
```

Expand Down
4 changes: 3 additions & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,9 @@ Scope — format and storage integrity:
The format is frozen for the golden-file corpus (#391).
- Index `verify` and public `rebuild_indexes()` (#373; done on `v3`)
- Net-assert on v8 keys before resolving facts, for O(live) point queries (#379; done on `v3`)
- One current valid-time window per `(e, a, v)`: a later assertion replaces the earlier window, and two windows of one fact in one transaction are `API-011` (#435; done on `v3`)
- One current valid-time window per `(e, a, v)`: a later assertion replaces the earlier window, and two windows of one fact in one `transact` are `API-011` (#435; done on `v3`)
- Empty or inverted valid-time windows rejected with `API-019` on every write path (#436; done on `v3`)
- In a `WriteTransaction`, the last statement that writes a fact decides it, so reads inside the transaction match the commit; `LogWriter` keeps the same rules, with `API-020` for an assertion and retraction of one fact in one transaction (#477; done on `v3`)
- `btree_page` fuzz target that reaches node decoding (#375)

Scope — query API:
Expand Down
48 changes: 43 additions & 5 deletions docs/ERROR_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,14 +168,16 @@ with no `CodedError` anywhere in its chain.
| API-008 | Function registry lock poisoned | Database API |
| API-009 | WAL not initialized | Database API |
| API-010 | Query with bind slots passed to execute() | Database API |
| API-011 | Two valid-time windows for one fact in one transaction | Database API |
| API-011 | Two valid-time windows for one fact in one transact | Database API |
| API-012 | Non-query command passed to query() | Database API |
| API-013 | Checkpoint while a fact log is open | Database API |
| API-014 | Write on a read-only handle | Database API |
| API-015 | Log writer record out of transaction order | Database API |
| API-016 | Log writer record with a second tx_id for one transaction | Database API |
| API-017 | Invalid argument passed through a language binding | Database API |
| API-018 | Call on a closed binding object | Database API |
| API-019 | Empty or inverted valid-time window | Database API |
| API-020 | Assertion and retraction of one fact in one log-writer transaction | Database API |
| INT-000 | Unclassified internal error | Internal |
| INT-001 | WriteTransaction already in progress on this thread | Internal |
| INT-002 | Invalid entity (API layer) | Internal |
Expand Down Expand Up @@ -2476,19 +2478,19 @@ let result = pq.execute(&[

**Scenario**: A query copied from the prepared-query docs is pasted into the REPL or passed to `db.execute()` unchanged.

### API-011 Two valid-time windows for one fact in one transaction
### API-011 Two valid-time windows for one fact in one transact

**Error text**: `one transaction asserts the same value of {} with two valid-time windows; assert it once with its final window`
**Error text**: `one (transact ...) asserts the same value of {} with two valid-time windows; assert it once with its final window`

**Cause**: One transaction asserts the same entity, attribute and value twice with different `:valid-from`/`:valid-to` bounds. At any transaction time a fact has exactly one current valid-time window, and the window of the latest transaction applies. Two windows in the same transaction leave no latest one, so the transaction is rejected and nothing is written. The `{}` is the attribute. This can come from one `(transact ...)` with per-fact bounds, from several `execute()` calls inside one `WriteTransaction`, or from `LogWriter::append` records of one transaction (that record alone is rejected).
**Cause**: One `(transact ...)` asserts the same entity, attribute and value twice with different `:valid-from`/`:valid-to` bounds. At any transaction time a fact has exactly one current valid-time window. Within one statement neither window comes later, so the statement is rejected and nothing is written. The `{}` is the attribute. Identical repeats are allowed and stored once. Inside a `WriteTransaction`, `tx.execute()` rejects the statement and stages nothing; the transaction stays usable. Across several statements of one `WriteTransaction` there is no error: the last statement that writes a fact decides it (#477). `LogWriter::append` returns this code for a record that asserts a fact the open transaction already asserts with another window; that record alone is rejected.

**Resolution**:
- Assert the fact once per transaction, with the window it should have.
- To change a window later (close it, extend it, reopen it), `transact` the same fact again with the new bounds. The earlier window stays visible through `:as-of`.
- A fact that was true over two separate periods cannot have both windows live at once. Model each period as its own entity (for example an `:employment` entity with `:employment/salary`), or keep the latest period current and read earlier ones with `:as-of`.

```clojure
;; Rejected: two windows for [:alice :salary 100000] in one transaction
;; Rejected: two windows for [:alice :salary 100000] in one transact
(transact [[:alice :salary 100000 {:valid-from "2020-01-01" :valid-to "2022-01-01"}]
[:alice :salary 100000 {:valid-from "2024-01-01"}]])

Expand Down Expand Up @@ -2604,6 +2606,42 @@ let mut log = src.fact_log(&FactFilter::new())?;

**Scenario**: A loader calls `finish()` in a `finally` block after it already called it on success.

### API-019 Empty or inverted valid-time window

**Error text**: `the valid-time window of {} ends at or before it starts; :valid-to must be later than :valid-from (the transaction time when :valid-from is omitted)`

**Cause**: A `transact` gives a fact a valid-time window whose end is at or before its start, so the fact could never match a `:valid-at` query. The check uses the effective window, after defaults: the fact's own bound, else the transaction's, else `:valid-from` is the transaction time and `:valid-to` is forever. A `:valid-to` in the past with no `:valid-from` is therefore rejected too. The whole transaction is rejected, nothing is written and no transaction number is used. The `{}` is the attribute. Inside a `WriteTransaction`, `tx.execute()` checks each statement. `commit()` checks again with the commit time, so a `:valid-to` that passed while staging can still fail at commit. `LogWriter::append` rejects such a record the same way; that record alone is rejected. Files written before this check are read as they are.

**Resolution**:
- Give `:valid-to` a later instant than `:valid-from`.
- To record a fact that is no longer true, give it a `:valid-from` before its `:valid-to`, or `retract` it.
- Check that a per-fact `:valid-from`/`:valid-to` does not invert the transaction-level window it overrides.

```clojure
;; Rejected: ends before it starts, and an empty window
(transact {:valid-from "2026-08-21" :valid-to "2026-08-20"} [[:d/c :description "r"]])
(transact {:valid-from "2026-08-21" :valid-to "2026-08-21"} [[:d/c :description "r"]])
;; Rejected: valid-from defaults to now, which is after valid-to
(transact {:valid-to "2020-01-01"} [[:d/c :description "r"]])

;; Accepted
(transact {:valid-from "2019-01-01" :valid-to "2020-01-01"} [[:d/c :description "r"]])
```

**Scenario**: An import sets `:valid-to` from a source's end date but leaves out `:valid-from`, so past end dates produce windows that end before the transaction time.

### API-020 Assertion and retraction of one fact in one log-writer transaction

**Error text**: `one transaction both asserts and retracts the same value of {}; keep only the record that decides it`

**Cause**: `LogWriter::append` was given a record that asserts an entity, attribute and value the open transaction retracts, or retracts one it asserts. Records of one transaction carry one `tx_id` and have no order, so the pair does not say which write came last. A normal write never produces such a transaction: in a `WriteTransaction` the last statement that writes a fact decides it (#477). The record alone is rejected, and the writer stays usable. The `{}` is the attribute. A source written by Minigraf 2.x, or by v3 builds before #477, can hold such transactions; copying one fails at the first such record.

**Resolution**:
- Keep only the record that decides the fact. Readers of the source hide the fact when its newest transaction holds a retraction, so dropping the transaction's assertions of that fact keeps every query result the same.
- Repair the source's other old records the same way before appending them: a fact with two windows in one transaction (API-011) keeps the window it should have, and an assertion whose window ends before it starts (API-019) is dropped.

**Scenario**: An offline purge copies a database migrated from v2.x through `fact_log()` and `LogWriter`, and one of its transactions retracted and re-asserted a fact.

---

## INT — Internal Errors
Expand Down
4 changes: 2 additions & 2 deletions docs/TEST_COVERAGE.md

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
# Valid-time window validation and statement order in transactions — Design

**Issues:** #436, #477 (broadened), tracker #383
**Milestone:** v3.0.0, branch `v3` (no format change; writes that succeed today can fail)
**Builds on:** `2026-10-06-one-window-per-triple-design.md` (#435)

## 1. Rules

1. **Empty or inverted windows are rejected (#436).** A `transact` whose effective
window for any fact has `valid_to <= valid_from` fails with the new **API-019** and
writes nothing. "Effective" means after the defaults are applied: a fact's own
bound, else the transaction's, else `valid_from` = the transaction time and
`valid_to` = forever. So `(transact {:valid-to "2020-01-01"} ...)` is rejected,
because its `valid_from` is now.
2. **Within one `(transact [...])`**, the same `(e, a, v)` with two different windows is
still **API-011**. Identical repeats are allowed and stored once. Duplicate triples in
one `retract` are allowed.
3. **Across statements of one `WriteTransaction`**, the last statement that writes an
`(e, a, v)` decides it, whether it is a `transact` or a `retract` (#477). At commit,
every record of a triple from an earlier statement is dropped. One `tx_id` cannot tell
several versions of a fact apart, so the committed transaction holds each triple's
records from one statement only. Reads inside the transaction already follow statement
order, so they now agree with the committed result. This replaces API-011's
cross-statement rejection.
4. **`LogWriter` keeps the same rules.** Every file it builds satisfies rules 1–3, as if
written normally: an inverted window is API-019, a second window of one triple in one
transaction API-011, and an assertion plus a retraction of one triple in one
transaction (either order) the new **API-020**. Records of one transaction share a
`tx_id` and have no order, so the pair cannot say which came last; no normal write
produces it after rule 3. Each rejected record changes nothing, as with API-015 and
API-016. An unfiltered copy is identical to a source that keeps the rules. A source
written before them (v2.x via migration, earlier v3 builds) fails at the first such
record, and a middleware step must repair it (drop the assertions where a retraction
decides, keep one window, drop inverted windows).

`load_fact`, WAL replay and migration never check rules 1–3. Files that already contain
such data open and read as before: readers keep their tie rules (a retraction in the
group hides the triple; several windows are kept once each).

## 2. Changes

- `src/error.rs`, `docs/ERROR_REFERENCE.md`: **API-019** "Empty or inverted valid-time
window", text `the valid-time window of {} ends at or before it starts; :valid-to must
be later than :valid-from (the transaction time when :valid-from is omitted)`. API-011
is narrowed to one `(transact ...)`, and its text says so.
- `src/graph/storage.rs`: `check_valid_windows(facts, tx_id)` rejects any assertion whose
`valid_from` (`VALID_FROM_USE_TX_TIME` resolved to `tx_id`) is not before `valid_to`.
`FactStorage::transact`/`transact_batch` call it next to `check_one_window_per_triple`.
- `Minigraf::execute`: take `tx_id` before `allocate_tx_count`, then run both checks, so a
rejected transaction takes no `tx_count`. The same applies to the browser `apply_write`.
- `WriteTransaction::execute` (transact): run both checks on the statement (window
check against the staged `tx_id`) before staging it. A rejected statement stages
nothing, and the transaction stays usable.
- `WriteTransaction::commit`: keep each triple's records from its newest staged
statement only (the highest synthetic `tx_count`), take the commit `tx_id`, run
`check_valid_windows` on what is left (a default `valid_from` moves to the commit time,
so a `:valid-to` between staging and commit is caught here), then allocate the
`tx_count` and write.
- `src/log_writer.rs`: per open transaction, the window (or retraction) written for each
triple; API-019, API-011 and API-020 are checked before the record is loaded. New
**API-020** "Assertion and retraction of one fact in one log-writer transaction".

## 3. Tests

- Unit: `check_valid_windows` (inverted, empty, default `valid_from` against `tx_id`,
retractions ignored, forever); the commit collapse for each mix of statements.
- `tests/valid_time_window_test.rs`: API-019 from `execute` (both bounds, equal bounds,
`:valid-to` only in the past, a per-fact override inverting a transaction window),
from `WriteTransaction::execute`, and at commit, each taking no `tx_count`; old data
with an inverted window still opens. Retract-then-assert, assert-then-retract,
assert-then-assert with another window, and retract-then-retract inside one
`WriteTransaction` give the same rows before and after commit, in memory,
checkpointed and reopened. API-011 inside one statement of a `WriteTransaction` is
raised by `tx.execute`, and the transaction still commits its other statements.
- `tests/log_writer_test.rs`: API-019, API-011 and API-020 (both orders), each leaving
the writer usable and the output exact; duplicate retractions, identical repeats and
the same triple in the next transaction are accepted.
- `tests/model_based_test.rs`: the model applies rules 1–3 (rejections from `execute`
and `tx.execute`, last statement wins at commit) and checks reads inside a write
transaction against the committed result.
7 changes: 5 additions & 2 deletions src/browser/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -322,11 +322,14 @@ impl BrowserDb {
let (flush, result_json) = {
let mut inner = self.inner.borrow_mut();

// Before allocating: a rejected transaction takes no tx_count (#435).
// Before allocating: a rejected transaction takes no tx_count
// (#435, #436).
let tx_id = tx_id_now();
crate::graph::storage::check_valid_windows(&facts, tx_id)
.map_err(|e| JsValue::from_str(&e.to_string()))?;
crate::graph::storage::check_one_window_per_triple(&facts)
.map_err(|e| JsValue::from_str(&e.to_string()))?;
let tx_count = inner.fact_storage.allocate_tx_count();
let tx_id = tx_id_now();

let stamped: Vec<crate::graph::types::Fact> = facts
.into_iter()
Expand Down
Loading
Loading
0