8000 fix(parser): reject what the grammar rejects (#651) by adityamukho · Pull Request #659 · project-minigraf/minigraf · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- **Every build reorders query clauses; the `wasm` feature does nothing** (#594, #596). The `wasm` Cargo feature kept patterns, `not` clauses and `or` branches in source order and turned off the `or` short-circuits (#250), for small browser datasets and a predictable order. With every answer independent of the plan (#594, #596, #606) it only made queries slower, and no first-party build enabled it (the browser package uses `browser`). The feature is kept, so builds that name it still resolve, but is deprecated and has no effect.
- **Aggregate and window answers no longer depend on the order of their input** (#547). Before, aggregates and window functions read their rows in the order the plan produced them, and several answers followed that order. Six cases changed. (1) `sum` over floats is exact: the values are added without rounding and the total is rounded once, so `1e16 + 1.0 + -1e16 + 1.0 + 0.5` gives `2.5` in every order (before, `1.5` or `2.5`). Window `sum` and `avg` likewise. (2) `sum` over integers saturates once, at the end: `i64::MAX + 1 + -1` gives `i64::MAX` (before, `i64::MAX - 1` or `i64::MAX` by order). (3) Window `sum` over integers adds them exactly; before they were added as floats, so a running total above 2^53 lost its low digits. (4) `min` / `max` of floats follow one order: `-0.0` before `0.0`, NaN after every other number (before, the first of two such values seen won). Window `min` / `max` use the same order over any values, so the minimum of keywords is the first in text order. (5) Window functions sort each partition by `:order-by` under one total order of values (numbers by numeric value, an integer before an equal float, then strings, booleans, refs, keywords, `nil`), so keywords, refs and booleans now sort (before, any two of them compared equal and kept their input order). Rows with equal `:order-by` values are peers: `rank` gives them the same rank, and `sum`, `count`, `min`, `max`, `avg` and registered window functions give every peer the value at the last peer, as SQL does; `row-number` numbers peers in the order of the other variables `:find` names. For example, with `?s` of 1.5, 5, 10, 10, `(sum ?s :over (:order-by ?s))` gave 1.5, 6.5, 16.5, 26.5, the 16.5 going to either tied row, and now gives 1.5, 6.5, 26.5, 26.5; ranking `:x :x :y :y` by `:order-by` gave ranks depending on the order the scan met them (1 2 3 4 when interleaved) and now gives 1 1 3 3. (6) In a query with aggregates, a window's argument, `:partition-by` and `:order-by` must be plain `:find` variables, or the query fails at parse time with the new `PRS-082`; before, any other variable read `nil` on every group row. The error a failing aggregate query reports is the first error, in `:find` order, of the first failing group in output order. A registered aggregate still receives its values in input order, which can depend on the plan; one whose result depends on that order may answer equivalent queries differently. (7) A group whose grouping variable holds floats that are equal as groups compare them (`-0.0` and `0.0`, or NaNs) returns the least of them in that order, so `-0.0` when the group holds it; before, it returned the first one the plan met.
- **Parse errors that were `INT-019` now carry their own `PRS` codes** (#643). The parser's long-tail syntax and structure errors (a bind slot with no name, a window without `:order-by`, a non-vector query, malformed `:with`, `:find`, valid-time map, retract fact, `not`/`not-join`/`or` body, pattern and rule) used one internal code, `INT-019`, with a `datalog parser: ` prefix. Each now has its own Parser-category code, `PRS-083` to `PRS-107`, documented in `docs/ERROR_REFERENCE.md`; the message is the old text without the prefix. A retract fact without exactly three elements was `INT-000` and is now `PRS-090`. Code that matches on `INT-019` (or `INT-000` for retract arity), or on the `Internal` category, for these errors must match the new codes. `INT-019` stays registered for parser invariants no input is known to reach.
- **The parser rejects what the grammar rejects** (#651). The grammar verifier (#643) found inputs the parser accepted that `tests/grammar/grammar.pest` and `docs/GRAMMAR.md` do not describe, several with wrong answers. Each is now an error:
- A value after a query option and its argument (`:as-of N`, `:valid-at "…"`, `:any-valid-time`, `:max-derived-facts N`, `:max-results N`) joined the section before the option: `[:find ?a :any-valid-time ?b :where …]` returned `?b` as a second column. It now fails with `PRS-061`.
- A `:where` list headed by an operator or a variable, `(< ?a ?b)` or `(?r ?a)`, was a call of a rule of that name and returned no rows. It now fails with `PRS-097`; a comparison is an expression clause, `[(< ?a ?b)]`.
- A rule named by a variable or an operator, `(rule [(?r ?a) …])`, fails with `PRS-105`, and a rule head argument that is not a variable, `(rule [(r ?a :c) …])`, with the new `PRS-114` (it bound the caller's argument to the constant; bind the variable in the body instead).
- An aggregate, window function or expression named by a variable, `(?f ?x)`, failed when the query ran (`QRY-010`, `QRY-007`); it now fails to parse with `PRS-025`, `PRS-029` or `PRS-063`.
- `(query [])` fails with the new `PRS-111`; `:find`, `:where` or `:with` with nothing after it (an empty `:find` returned one empty row) with the new `PRS-112`; a keyword the query vector does not know, such as `:limit`, with the new `PRS-113` (it was ignored when another keyword followed, and `PRS-061` otherwise).
- A string with no closing quote fails with the new `PRS-109` (it ran to the end of the input), and a `:` with no name after it, such as `:+`, with the new `PRS-110` (it was the keyword `:`).

A lone `-` stays the subtraction operator; the grammar now lists it with the other operators. The verifier's #651 entries are removed.

### Added

Expand Down Expand Up @@ -105,7 +114,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `tests/snapshot_atomicity_test.rs` races a writer against a reader: a query must see each transaction whole, whether written with `execute` or a `WriteTransaction` (#554), and must read one state for all its patterns (#555), in memory and on file. It also runs the v2.x checkpoint race from #558, which never failed on v3: copy-on-write checkpoints do not overwrite pages a published reader uses.
- **The `datalog_eval` fuzz guard counts partial bindings** (#597). The guard skipped inputs whose patterns join to more than 50,000 rows, counting only the rows after the last pattern. A `not-join` body still runs on the executor pattern by pattern in text order, so a 120-patt 8000 ern body built hundreds of thousands of partial bindings before its last pattern matched nothing, and ran past the 60-second timeout. The guard now counts the bindings after every pattern. On the `v3` corpus (4,783 inputs) coverage drops by 7 of 9,719 edges. The input is added as a seed, as is the `btree_page` input of the cursor fix above.

- **Parser vs grammar verifier (#643).** `src/query/datalog/parser/verifier.rs` checks, for every generated input, that the parser rejects what `tests/grammar/grammar.pest` rejects, and accepts what it accepts unless the error code is semantic (a listed code with a witness pest accepts). It runs every string of 1 to 3 characters over 102 characters (ASCII, `\t \n \r`, four non-ASCII stand-ins) through `parse_edn`, and every bracket-balanced sequence of up to `MINIGRAF_GRAMMAR_BOUND` (default 3) of 32 tokens in 15 command templates through `parse_datalog_command`, also with no separators up to 3 tokens. Deterministic; about 20 s by default and 3.5 min at bound 4 on a 16-thread machine. The drift it found is listed in `KNOWN_DRIFT` by cause (comments and non-ASCII letters #650, since documented: see below; empty sections, ignored keywords, rule and call heads, a lone `:` or `-`, unterminated strings #651; a fact of more than four elements losing elements, #649, since fixed); each fix removes its entry. `grammar.pest` now says what the parser means: a fact is exactly `[e a v]` plus an optional valid-time map and a retracted fact exactly `[e a v]`; `not`, `not-join`, `or`, `or-join` (and `and` inside an `or`) are never rule names; literal keywords end at a word boundary (`:find?x` is one keyword).
- **Parser vs grammar verifier (#643).** `src/query/datalog/parser/verifier.rs` checks, for every generated input, that the parser rejects what `tests/grammar/grammar.pest` rejects, and accepts what it accepts unless the error code is semantic (a listed code with a witness pest accepts). It runs every string of 1 to 3 characters over 102 characters (ASCII, `\t \n \r`, four non-ASCII stand-ins) through `parse_edn`, and every bracket-balanced sequence of up to `MINIGRAF_GRAMMAR_BOUND` (default 3) of 32 tokens in 15 command templates through `parse_datalog_command`, also with no separators up to 3 tokens. Deterministic; about 20 s by default and 3.5 min at bound 4 on a 16-thread machine. The drift it found is listed in `KNOWN_DRIFT` by cause (comments and non-ASCII letters #650, since documented: see below; empty sections, ignored keywords, rule and call heads, a lone `:` or `-`, unterminated strings #651, since fixed; a fact of more than four elements losing elements, #649, since fixed); each fix removes its entry. `grammar.pest` now says what the parser means: a fact is exactly `[e a v]` plus an optional valid-time map and a retracted fact exactly `[e a v]`; `not`, `not-join`, `or`, `or-join` (and `and` inside an `or`) are never rule names; literal keywords end at a word boundary (`:find?x` is one keyword).
- **The grammar documents comments and non-ASCII identifiers** (#650). The parser has always accepted both; `tests/grammar/grammar.pest` and `docs/GRAMMAR.md` now say so. A `;` starts a comment that runs to the end of the line and is skipped like whitespace. Symbols, variables, keywords and bind slots take Unicode letters and numerals (`:person/名前`, `?ñ`), as Rust's `char::is_alphabetic` and `char::is_numeric` define them; a symbol starts with a letter or `_`, and `-` followed by a numeral starts a number, not a symbol. Nothing the parser accepts or rejects changes.

### Documentation
Expand Down
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ cargo run < demos/demo_negation.txt

3. **`src/query/datalog/`** — Datalog engine
- `parser.rs`: EDN/Datalog parser — `transact`, `retract`, `query`, `rule`, `:as-of`, `:valid-at`, `not`, `not-join`; `resolve_pseudo_sources` names each per-fact pseudo-attribute's source (nearest earlier same-entity pattern, lexical scope) and its hidden metadata variable `?#<n>.<attr>`, else PRS-081 (#594)
- `parser/verifier.rs`: test-only (not wasm32) bounded-exhaustive check of the parser against `tests/grammar/grammar.pest` (#643): every 1–3 character string (`parse_edn`) and every balanced token filler up to `MINIGRAF_GRAMMAR_BOUND` (default 3) in 15 command templates (`parse_datalog_command`); pest rejects ⇒ parser rejects, pest accepts ⇒ parser accepts or raises a `SEMANTIC` code; today's drift is `KNOWN_DRIFT`, one entry per cause, removed by its fix (#651; #649, #650 fixed)
- `parser/verifier.rs`: test-only (not wasm32) bounded-exhaustive check of the parser against `tests/grammar/grammar.pest` (#643): every 1–3 character string (`parse_edn`) and every balanced token filler up to `MINIGRAF_GRAMMAR_BOUND` (default 3) in 15 command templates (`parse_datalog_command`); pest rejects ⇒ parser rejects, pest accepts ⇒ parser acce 8000 pts or raises a `SEMANTIC` code; today's drift is `KNOWN_DRIFT`, one entry per cause, removed by its fix (empty: #649, #650, #651 fixed)
- `executor.rs`: Query executor — temporal filter (tx-time → net-assert → valid-time), not/not-join post-filter
- `matcher.rs`: Pattern matching with variable unification; `edn_to_value`, `edn_to_entity_id`
- `aggregate.rs`: `Agg` (one aggregate of one group) and `Running` (a window's running value), shared by the executor and the engine; exact sums (`i128` integers saturated once, fixed-point floats rounded once), `value_order` (one total order of values for window sorts, peers, `rank` and `min`/`max`), `named_vars` (a window's tie-break variables), `windows` (each window's value per row: partitions, sort, peers) (#547)
Expand Down Expand Up @@ -206,7 +206,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are

## Test Coverage

**1662 tests** (1652 passing, 10 ignored; unit + integration + doc).
**1672 tests** (1662 passing, 10 ignored; unit + integration + doc).
See `docs/TEST_COVERAGE.md` for the full per-file breakdown.

**Testing conventions** — see the Testing Conventions section below before writing any tests.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit

```bash
cargo run # interactive Datalog REPL
cargo test # run 1662 tests
cargo test # run 1672 tests
cargo run < demos/demo_recursive.txt # recursive rules demo
```

Expand Down
Loading
Loading
0