8000 fix(parser): reject a fact with more than four elements, PRS-108 (#649) by adityamukho · Pull Request #654 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **All patterns of one query read the same state of the database** (#555). A query whose patterns name entities or attributes read each of them with its own lookup, each under its own lock, so a transaction that committed between two lookups was seen by some patterns and not others, and the query could return rows that match no state of the database (an entity with one attribute but not another that was always written with it). The lookups now run under one lock. Stored data was never affected. Affected every release since v1.1.0.
- **A query never sees a checkpoint half published** (#543). A checkpoint cleared the in-memory pending facts and installed the new committed index as two steps, so a query between them could miss the facts being checkpointed or, with the opposite order, see them twice. Both now change under one lock. Every read of one query, cursor or fact log is taken from one snapshot of the database, which holds the committed generation it read until it is dropped. This affected only v3.0.0 development builds; in released versions the checkpoint race is #558, fixed separately.
- **A damaged index tree whose node points back up fails the query with `INT-049` instead of never finishing** (#597). A scan resumes each batch after the last key it read. When a corrupt internal node led back to a leaf already read, the scan read the same keys again on every batch and never ended, holding more memory each time; the nightly fuzz run found it as an out-of-memory. The B+tree cursor now requires keys to increase from one leaf to the next and never to fall below its start or the key it was last moved to, and refuses an empty leaf below an internal node; any of these fails with `INT-049`. `verify()` already reported such a tree.
- **A fact with more than four elements is rejected** (#649). `transact` read a fact's 4th element as its optional valid-time map and ignored everything after it, so `[[:alice :person/age 1 {} 99 :junk]]` stored `[:alice :person/age 1]` and dropped the rest without an error; two facts run together by a misplaced bracket lost one. A fact of more than four elements now fails with the new `PRS-108`, and the transaction writes nothing. A fact of five or more elements whose 4th is not a map now gets `PRS-108` instead of `PRS-047`. Present since v0.4.0; found by the grammar verifier (#643), whose `FactArity` drift entry is removed.

### Internal

Expand Down Expand Up @@ -104,7 +105,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `tests/snapshot_atomicity_test.rs` races a writer against a reader: a query must see each transaction whole, whether written with `execute` or a `WriteTransaction` (#554), and must read one state for all its patterns (#555), in memory and on file. It also runs the v2.x checkpoint race from #558, which never failed on v3: copy-on-write checkpoints do not overwrite pages a published reader uses.
- **The `datalog_eval` fuzz guard counts partial bindings** (#597). The guard skipped inputs whose patterns join to more than 50,000 rows, counting only the rows after the last pattern. A `not-join` body still runs on the executor pattern by pattern in text order, so a 120-pattern body built hundreds of thousands of partial bindings before its last pattern matched nothing, and ran past the 60-second timeout. The guard now counts the bindings after every pattern. On the `v3` corpus (4,783 inputs) coverage drops by 7 of 9,719 edges. The input is added as a seed, as is the `btree_page` input of the cursor fix above.

- **Parser vs grammar verifier (#643).** `src/query/datalog/parser/verifier.rs` checks, for every generated input, that the parser rejects what `tests/grammar/grammar.pest` rejects, and accepts what it accepts unless the error code is semantic (a listed code with a witness pest accepts). It runs every string of 1 to 3 characters over 102 characters (ASCII, `\t \n \r`, four non-ASCII stand-ins) through `parse_edn`, and every bracket-balanced sequence of up to `MINIGRAF_GRAMMAR_BOUND` (default 3) of 32 tokens in 15 command templates through `parse_datalog_command`, also with no separators up to 3 tokens. Deterministic; about 20 s by default and 3.5 min at bound 4 on a 16-thread machine. The drift it found is listed in `KNOWN_DRIFT` by cause (comments and non-ASCII letters #650; empty sections, ignored keywords, rule and call heads, a lone `:` or `-`, unterminated strings #651; a fact of more than four elements losing elements, #649); each fix removes its entry. `grammar.pest` now says what the parser means: a fact is exactly `[e a v]` plus an optional valid-time map and a retracted fact exactly `[e a v]`; `not`, `not-join`, `or`, `or-join` (and `and` inside an `or`) are never rule names; literal keywords end at a word boundary (`:find?x` is one keyword).
- **Parser vs grammar verifier (#643).** `src/query/datalog/parser/verifier.rs` checks, for every generated input, that the parser rejects what `tests/grammar/grammar.pest` rejects, and accepts what it accepts unless the error code is semantic (a listed code with a witness pest accepts). It runs every string of 1 to 3 characters over 102 characters (ASCII, `\t \n \r`, four non-ASCII stand-ins) through `parse_edn`, and every bracket-balanced sequence of up to `MINIGRAF_GRAMMAR_BOUND` (default 3) of 32 tokens in 15 command templates through `parse_datalog_command`, also with no separators up to 3 tokens. Deterministic; about 20 s by default and 3.5 min at bound 4 on a 16-thread machine. The drift it found is listed in `KNOWN_DRIFT` by cause (comments and non-ASCII letters #650; empty sections, ignored keywords, rule and call heads, a lone `:` or `-`, unterminated strings #651; a fact of more than four elements losing elements, #649, since fixed); each fix removes its entry. `grammar.pest` now says what the parser means: a fact is exactly `[e a v]` plus an optional valid-time map and a retracted fact exactly `[e a v]`; `not`, `not-join`, `or`, `or-join` (and `and` inside an `or`) are never rule names; literal keywords end at a word boundary (`:find?x` is one keyword).

### Documentation

Expand Down
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ cargo run < demos/demo_negation.txt

3. **`src/query/datalog/`** — Datalog engine
- `parser.rs`: EDN/Datalog parser — `transact`, `retract`, `query`, `rule`, `:as-of`, `:valid-at`, `not`, `not-join`; `resolve_pseudo_sources` names each per-fact pseudo-attribute's source (nearest earlier same-entity pattern, lexical scope) and its hidden metadata variable `?#<n>.<attr>`, else PRS-081 (#594)
- `parser/verifier.rs`: test-only (not wasm32) bounded-exhaustive check of the parser against `tests/grammar/grammar.pest` (#643): every 1–3 character string (`parse_edn`) and every balanced token filler up to `MINIGRAF_GRAMMAR_BOUND` (default 3) in 15 command templates (`parse_datalog_command`); pest rejects ⇒ parser rejects, pest accepts ⇒ parser accepts or raises a `SEMANTIC` code; today's drift is `KNOWN_DRIFT`, one entry per cause, removed by its fix (#649–#651)
- `parser/verifier.rs`: test-only (not wasm32) bounded-exhaustive check of the parser against `tests/grammar/grammar.pest` (#643): every 1–3 character string (`parse_edn`) and every balanced token filler up to `MINIGRAF_GRAMMAR_BOUND` (default 3) in 15 command templates (`parse_datalog_command`); pest rejects ⇒ parser rejects, pest accepts ⇒ parser accepts or raises a `SEMANTIC` code; today's drift is `KNOWN_DRIFT`, one entry per cause, removed by its fix (#650, #651; #649 fixed)
- `executor.rs`: Query executor — temporal filter (tx-time → net-assert → valid-time), not/not-join post-filter
- `matcher.rs`: Pattern matching with variable unification; `edn_to_value`, `edn_to_entity_id`
- `aggregate.rs`: `Agg` (one aggregate of one group) and `Running` (a window's running value), shared by the executor and the engine; exact sums (`i128` integers saturated once, fixed-point floats rounded once), `value_order` (one total order of values for window sorts, peers, `rank` and `min`/`max`), `named_vars` (a window's tie-break variables), `windows` (each window's value per row: partitions, sort, peers) (#547)
Expand Down Expand Up @@ -206,7 +206,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are

## Test Coverage

**1659 tests** (1649 passing, 10 ignored; unit + integration + doc).
**1661 tests** (1651 passing, 10 ignored; unit + integration + doc).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit

```bash
cargo run # interactive Datalog REPL
cargo test # run 1659 tests
cargo test # run 1661 tests
cargo run < demos/demo_recursive.txt # recursive rules demo
```

Expand Down
22 changes: 22 additions & 0 deletions docs/ERROR_REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,7 @@ with no `CodedError` anywhere in its chain.
| PRS-105 | Rule head without a predicate name | Parser |
| PRS-106 | Rule body clause is not a vector or list | Parser |
| PRS-107 | Rule body has no clause | Parser |
| PRS-108 | Fact has more than 4 elements | Parser |
| QRY-001 | Invalid entity | Query Execution |
| QRY-002 | Attribute must be a keyword | Query Execution |
| QRY-003 | Cannot transact a pseudo-attribute | Query Execution |
Expand Down Expand Up @@ -2013,6 +2014,27 @@ $<4097-char-slot-name>
```
*Add a clause, e.g. `[?a :edge ?b]`*

### PRS-108 Fact has more than 4 elements

**Error text**: `Fact must have at most 4 elements (E A V and an optional map), got {}`

**Cause**: A fact vector in `transact` has more than four elements. A fact is
`[E A V]` or `[E A V {:valid-from ... :valid-to ...}]`. Through v2.0.8, a
fact whose 4th element was a map had every element after the map dropped
without an error (#649), and one whose 4th element was not a map was rejected
with `PRS-047`.

**Resolution**:
- Check the brackets: two facts run together,
`(transact [[e a v {} e2 a2 v2]])`, are two vectors in the facts vector,
`(transact [[e a v {}] [e2 a2 v2]])`.

**Example**:
```datalog
(transact [[:alice :person/age 1 {} 99 :junk]])
```
*Remove `99 :junk`, or make it its own fact*

---

## QRY — Query Execution Errors
Expand Down
5 changes: 3 additions & 2 deletions docs/TEST_COVERAGE.md

Large diffs are not rendered by default.

9 changes: 9 additions & 0 deletions src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,7 @@ pub(crate) enum ErrorCode {
Prs105,
Prs106,
Prs107,
Prs108,
Qry001,
Qry002,
Qry003,
Expand Down Expand Up @@ -954,6 +955,12 @@ pub(crate) const REGISTRY: &[(ErrorCode, &str, &str, ErrorCategory)] = &[
"Rule must have at least one pattern or rule invocation in body",
ErrorCategory::Parser,
),
(
ErrorCode::Prs108,
"PRS-108",
"Fact must have at most 4 elements (E A V and an optional map), got {}",
ErrorCategory::Parser,
),
(
ErrorCode::Qry001,
"QRY-001",
Expand Down Expand Up @@ -2200,6 +2207,7 @@ mod tests {
| ErrorCode::Prs105
| ErrorCode::Prs106
| ErrorCode::Prs107
| ErrorCode::Prs108
| ErrorCode::Qry001
| ErrorCode::Qry002
| ErrorCode::Qry003
Expand Down Expand Up @@ -2453,6 +2461,7 @@ mod tests {
ErrorCode::Prs105,
ErrorCode::Prs106,
ErrorCode::Prs107,
ErrorCode::Prs108,
ErrorCode::Qry001,
ErrorCode::Qry002,
ErrorCode::Qry003,
Expand Down
3 changes: 3 additions & 0 deletions src/query/datalog/parser.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1030,6 +1030,9 @@ fn parse_transact(elements: &[EdnValue]) -> Result<DatalogCommand> {
if fact_vec.len() < 3 {
bail_coded!(ErrorCode::Prs046, fact_vec.len());
}
if fact_vec.len() > 4 {
bail_coded!(ErrorCode::Prs108, fact_vec.len());
}

// SAFETY: len >= 3 check above guarantees indices 0, 1, 2 exist
#[allow(clippy::indexing_slicing)]
Expand Down
11 changes: 0 additions & 11 deletions src/query/datalog/parser/verifier.rs
Original file line number Diff line number Diff line change
Expand Up @@ -255,8 +255,6 @@ enum Cause {
/// A rule head whose name is not a plain symbol or whose argument is not a
/// variable.
RuleHead,
/// A fact `[e a v {…} x …]`: the parser drops everything after the map.
FactArity,
}

impl Cause {
Expand Down Expand Up @@ -350,11 +348,6 @@ const KNOWN_DRIFT: &[Drift] = &[
issue: 651,
min_k: 1,
},
Drift {
cause: Cause::FactArity,
issue: 649,
min_k: 3,
},
];

/// Characters the parser reads as part of a symbol after its first.
Expand Down Expand Up @@ -405,10 +398,6 @@ fn structural_cause(text: &str) -> Option<Cause> {
match cmd.first()? {
EdnValue::Symbol(s) if s == "query" => query_cause(arg?.as_vector()?),
EdnValue::Symbol(s) if s == "rule" => rule_cause(arg?.as_vector()?),
EdnValue::Symbol(s) if s == "transact" => cmd.last()?.as_vector()?.iter().find_map(|f| {
let f = f.as_vector()?;
(f.len() > 4 && f[3].is_map()).then_some(Cause::FactArity)
}),
_ => None,
}
}
Expand Down
< D730 div class="file-header d-flex flex-md-row flex-column flex-md-items-center file-header--expandable js-file-header js-skip-tagsearch sticky-file-header secondary-sticky-header" data-path="tests/error_codes_prs_test.rs" data-short-path="f89d649" data-anchor="diff-f89d64938596af0636ca951bc0e2d6123a7ad675c794832f6022e25c084c04b2" data-file-type=".rs" data-file-deleted="false" >
35 changes: 35 additions & 0 deletions tests/error_codes_prs_test.rs
Original file line number Diff line number Diff line change
Expand Up @@ -720,3 +720,38 @@ fn prs_083_to_107_reach_through_prepare() {
assert_prepare_code(input, code);
}
}

// ── PRS-108: a fact with more than four elements (#649) ─────────────────────

#[test]
fn prs_108_fact_with_more_than_four_elements() {
assert_code_and_message(
"(transact [[:alice :person/age 1 {} 99 :junk]])",
"PRS-108",
"got 6",
);
assert_code_and_message(
r#"(transact [[:alice :name "Alice" "bogus" 7]])"#,
"PRS-108",
"got 5",
);
}

#[test]
fn prs_108_rejects_the_whole_transaction() {
// Before #649 the elements after the map were dropped and the fact stored.
let db = db();
db.execute(r#"(transact [[:bob :person/age 2] [:alice :person/age 1 {} 99 :junk]])"#)
.expect_err("a 6-element fact must be rejected");
let result = db
.execute("(query [:find ?e ?v :where [?e :person/age ?v]])")
.expect("query must run");
let minigraf::QueryResult::QueryResults { results, .. } = result else {
panic!("expected query results");
};
assert_eq!(
results.len(),
0,
"no fact of the rejected transaction is stored"
);
}
Loading
0