adityamukho · Oct 20, 2026 · Oct 20, 2026
@@ -13,7 +13,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Tests
 
-- **Parser vs grammar verifier (#643), backported from v3.0.0.** `src/query/datalog/parser/verifier.rs` checks, for every generated input, that the parser rejects what `tests/grammar/grammar.pest` rejects, and accepts what it accepts unless the error code is semantic (a listed code with a witness pest accepts). It runs every string of 1 to 3 characters over 102 characters (ASCII, `\t \n \r`, four non-ASCII stand-ins) through `parse_edn`, and every bracket-balanced sequence of up to `MINIGRAF_GRAMMAR_BOUND` (default 3) of 32 tokens in 15 command templates through `parse_datalog_command`, also with no separators up to 3 tokens. Deterministic; about 40 s by default in a debug build. On v2.x `INT-019` still mixes syntax and semantic parse errors (v3.0.0 splits it into `PRS-083`–`PRS-107`), so it counts as semantic here, and a syntax error coded `INT-019` that pest accepts is not caught (v3.0.0 checks that too). The drift it found is the same as on v3 and is listed in `KNOWN_DRIFT` by cause (comments and non-ASCII letters #650; empty sections, ignored keywords, rule and call heads, a lone `:` or `-`, unterminated strings #651; a fact of more than four elements, #649, is fixed above). `grammar.pest` now says what the parser means: a fact is exactly `[e a v]` plus an optional valid-time map and a retracted fact exactly `[e a v]`; `not`, `not-join`, `or`, `or-join` (and `and` inside an `or`) are never rule names; literal keywords end at a word boundary (`:find?x` is one keyword). A nightly workflow, `grammar-verifier.yml`, runs it at bound 4 on `main` and on `v3`, and opens or comments on an issue when it fails.
+- **Parser vs grammar verifier (#643), backported from v3.0.0.** `src/query/datalog/parser/verifier.rs` checks, for every generated input, that the parser rejects what `tests/grammar/grammar.pest` rejects, and accepts what it accepts unless the error code is semantic (a listed code with a witness pest accepts). It runs every string of 1 to 3 characters over 102 characters (ASCII, `\t \n \r`, four non-ASCII stand-ins) through `parse_edn`, and every bracket-balanced sequence of up to `MINIGRAF_GRAMMAR_BOUND` (default 3) of 32 tokens in 15 command templates through `parse_datalog_command`, also with no separators up to 3 tokens. Deterministic; about 40 s by default in a debug build. On v2.x `INT-019` still mixes syntax and semantic parse errors (v3.0.0 splits it into `PRS-083`–`PRS-107`), so it counts as semantic here, and a syntax error coded `INT-019` that pest accepts is not caught (v3.0.0 checks that too). The drift it found is the same as on v3 and is listed in `KNOWN_DRIFT` by cause (comments and non-ASCII letters #650, since documented: see below; empty sections, ignored keywords, rule and call heads, a lone `:` or `-`, unterminated strings #651; a fact of more than four elements, #649, is fixed above).
+- **The grammar documents comments and non-ASCII identifiers** (#650), backported from v3.0.0. The parser has always accepted both; `tests/grammar/grammar.pest` now says so. A `;` starts a comment that runs to the end of the line and is skipped like whitespace. Symbols, variables, keywords and bind slots take Unicode letters and numerals (`:person/名前`, `?ñ`), as Rust's `char::is_alphabetic` and `char::is_numeric` define them; a symbol starts with a letter or `_`, and `-` followed by a numeral starts a number, not a symbol. Nothing the parser accepts or rejects changes. `grammar.pest` now says what the parser means: a fact is exactly `[e a v]` plus an optional valid-time map and a retracted fact exactly `[e a v]`; `not`, `not-join`, `or`, `or-join` (and `and` inside an `or`) are never rule names; literal keywords end at a word boundary (`:find?x` is one keyword). A nightly workflow, `grammar-verifier.yml`, runs it at bound 4 on `main` and on `v3`, and opens or comments on an issue when it fails.
 
 ## v2.0.8 — 2026-10-20
 

@@ -118,7 +118,7 @@ cargo run < demos/demo_negation.txt
 
 3. **`src/query/datalog/`** — Datalog engine
    - `parser.rs`: EDN/Datalog parser — `transact`, `retract`, `query`, `rule`, `:as-of`, `:valid-at`, `not`, `not-join`
-   - `parser/verifier.rs`: test-only (not wasm32) bounded-exhaustive check of the parser against `tests/grammar/grammar.pest` (#643): every 1–3 character string (`parse_edn`) and every balanced token filler up to `MINIGRAF_GRAMMAR_BOUND` (default 3) in 15 command templates (`parse_datalog_command`); pest rejects ⇒ parser rejects, pest accepts ⇒ parser accepts or raises a `SEMANTIC` code (`INT-019` counts as semantic on v2.x); today's drift is `KNOWN_DRIFT`, one entry per cause, removed by its fix (#650, #651); `grammar-verifier.yml` runs bound 4 nightly
+   - `parser/verifier.rs`: test-only (not wasm32) bounded-exhaustive check of the parser against `tests/grammar/grammar.pest` (#643): every 1–3 character string (`parse_edn`) and every balanced token filler up to `MINIGRAF_GRAMMAR_BOUND` (default 3) in 15 command templates (`parse_datalog_command`); pest rejects ⇒ parser rejects, pest accepts ⇒ parser accepts or raises a `SEMANTIC` code (`INT-019` counts as semantic on v2.x); today's drift is `KNOWN_DRIFT`, one entry per cause, removed by its fix (#651; #650 fixed); `grammar-verifier.yml` runs bound 4 nightly
    - `executor.rs`: Query executor — temporal filter (tx-time → net-assert → valid-time), not/not-join post-filter
    - `matcher.rs`: Pattern matching with variable unification; `edn_to_value`, `edn_to_entity_id`
    - `evaluator.rs`: `RecursiveEvaluator` (semi-naive), `StratifiedEvaluator` (repeats a stratum whose rules recurse through a `not`/`or` rule until it derives nothing new, #633), `evaluate_not_join`

@@ -22,7 +22,7 @@ For v2.0.5, torn-WAL-tail tests (#513) check that opening a WAL writer cuts off
 
 `tests/error_codes_prs_test.rs` checks that a fact of five or six elements, with or without a map 4th, fails with `PRS-108`, and that a transaction holding one stores none of its facts (#649).
 
-Since v2.0.8, `src/query/datalog/parser/verifier.rs` (4 tests, backported from v3.0.0; #643) checks the parser against `tests/grammar/grammar.pest` on about 1.07M lexical strings and 555K structural inputs at the default bound (7.6M at `MINIGRAF_GRAMMAR_BOUND=4`, run nightly by `grammar-verifier.yml`): pest rejects ⇒ the parser rejects, pest accepts ⇒ the parser accepts or raises a `SEMANTIC` code with a witness (`INT-019` counts as semantic on v2.x). Known drift is listed by cause in `KNOWN_DRIFT` (#650, #651). `tests/grammar_conformance.rs` now classes a fact with too few elements and a retract of the wrong arity as syntax errors, since `grammar.pest`'s fact is exact.
+Since v2.0.8, `src/query/datalog/parser/verifier.rs` (4 tests, backported from v3.0.0; #643) checks the parser against `tests/grammar/grammar.pest` on about 1.07M lexical strings and 555K structural inputs at the default bound (7.6M at `MINIGRAF_GRAMMAR_BOUND=4`, run nightly by `grammar-verifier.yml`): pest rejects ⇒ the parser rejects, pest accepts ⇒ the parser accepts or raises a `SEMANTIC` code with a witness (`INT-019` counts as semantic on v2.x). Known drift is listed by cause in `KNOWN_DRIFT` (#651). `tests/grammar_conformance.rs` now classes a fact with too few elements and a retract of the wrong arity as syntax errors, since `grammar.pest`'s fact is exact.
 
 Not counted above: the `BrowserDb` tests run only on wasm32 with `wasm-bindgen-test` in a browser. Two of them check that the IndexedDB flush returns every page a write dirtied, and that a dirty page that cannot be read is an error rather than left out.
 

@@ -221,12 +221,8 @@ fn is_semantic(code: &str) -> bool {
 #[derive(Clone, Copy, PartialEq, Eq, Debug)]
 enum Cause {
     // Lexical.
-    /// `;` starts a line comment in the parser; the grammar has no comments.
-    Comment,
     /// A string with no closing quote reads to the end of the input.
     UnterminatedString,
-    /// A non-ASCII letter or digit in a symbol or keyword (the grammar is ASCII).
-    UnicodeLetter,
     /// `:` with no keyword character after it is the keyword `:`.
     BareColon,
     /// `-` with no word character after it is the symbol `-`.
@@ -257,11 +253,7 @@ enum Cause {
 impl Cause {
     fn level(self) -> Level {
         match self {
-            Cause::Comment
-            | Cause::UnterminatedString
-            | Cause::UnicodeLetter
-            | Cause::BareColon
-            | Cause::LoneMinus => Level::Lexical,
+            Cause::UnterminatedString | Cause::BareColon | Cause::LoneMinus => Level::Lexical,
             _ => Level::Structural,
         }
     }
@@ -275,21 +267,11 @@ struct Drift {
 }
 
 const KNOWN_DRIFT: &[Drift] = &[
-    Drift {
-        cause: Cause::Comment,
-        issue: 650,
-        min_k: 0,
-    },
     Drift {
         cause: Cause::UnterminatedString,
         issue: 651,
         min_k: 0,
     },
-    Drift {
-        cause: Cause::UnicodeLetter,
-        issue: 650,
-        min_k: 0,
-    },
     Drift {
         cause: Cause::BareColon,
         issue: 651,
@@ -370,14 +352,12 @@ fn lexical_cause(text: &str) -> Option<Cause> {
                     }
                 }
             }
-            ';' => return Some(Cause::Comment),
             ':' if !next.is_some_and(symbol_char) => return Some(Cause::BareColon),
             '-' if !next.is_some_and(symbol_char)
                 && !(i > 0 && (symbol_char(c[i - 1]) || matches!(c[i - 1], ':' | '#' | '$'))) =>
             {
                 return Some(Cause::LoneMinus);
             }
-            ch if !ch.is_ascii() && ch.is_alphanumeric() => return Some(Cause::UnicodeLetter),
             _ => {}
         }
         i += 1;
@@ -455,11 +435,14 @@ fn query_cause(items: &[EdnValue]) -> Option<Cause> {
 
 /// Whether pest's `plain_sym` reads all of `s`.
 fn is_plain_symbol(s: &str) -> bool {
-    let word = |c: char| c.is_ascii_alphanumeric() || matches!(c, '?' | '_' | '-' | '/');
     let mut chars = s.chars();
     match chars.next() {
-        Some(c) if c.is_ascii_alphabetic() || c == '_' => chars.all(word),
-        Some('-') => !chars.as_str().is_empty() && chars.all(word),
+        Some(c) if c.is_alphabetic() || c == '_' => chars.all(symbol_char),
+        Some('-') => {
+            !chars.as_str().is_empty()
+                && !chars.as_str().starts_with(char::is_numeric)
+                && chars.all(symbol_char)
+        }
         _ => false,
     }
 }

@@ -1,9 +1,13 @@
 // ── WHITESPACE (auto-skipped between every token) ─────────────────────────
 WHITESPACE = _{ " " | "\t" | "\n" | "\r" | "," }
 
+// Line comment: ";" to the end of the line, skipped like whitespace (#650).
+COMMENT = _{ ";" ~ (!"\n" ~ ANY)* }
+
 // ── WORD-BOUNDARY HELPER ──────────────────────────────────────────────────
-// Used in negative lookaheads to enforce keyword boundaries.
-word_cont = _{ ASCII_ALPHANUMERIC | "?" | "_" | "-" | "/" }
+// Used in negative lookaheads to enforce keyword boundaries. Letters and numerals are
+// Unicode's (Alphabetic; Nd, Nl, No), as the parser reads them (#650).
+word_cont = _{ ALPHABETIC | NUMBER | "?" | "_" | "-" | "/" }
 
 // Literal keywords: atomic, so that no whitespace is skipped before the
 // boundary check (":find?x" is one keyword, as the parser reads it).
@@ -52,7 +56,7 @@ string_lit = @{ "\"" ~ str_char* ~ "\"" }
 uuid_lit = { "#uuid" ~ string_lit }
 
 // Prepared-query bind slot: $name
-bind_slot = @{ "$" ~ (ASCII_ALPHANUMERIC | "_" | "-")+ }
+bind_slot = @{ "$" ~ (ALPHABETIC | NUMBER | "_" | "-")+ }
 
 // Keyword: starts with ":", followed by word constituents (incl. "?")
 keyword = @{ ":" ~ word_cont+ }
@@ -63,11 +67,12 @@ variable = @{ "?" ~ word_cont* }
 // Operator symbols used in expressions
 op_sym = @{ "<=" | ">=" | "!=" | "<" | ">" | "=" | "+" | "*" | "/" }
 
-// Plain symbol: letter or underscore start; or "-" followed by at least one word char.
+// Plain symbol: letter or underscore start; or "-" followed by at least one word char
+// that is not a numeral (after "-" the parser reads a numeral as a number).
 // Negative lookaheads prevent "true", "false", "nil" from matching as plain symbols.
 plain_sym = @{
-    !boolean_lit ~ !nil_lit ~ (ASCII_ALPHA | "_") ~ word_cont*
-    | "-" ~ word_cont+
+    !boolean_lit ~ !nil_lit ~ (ALPHABETIC | "_") ~ word_cont*
+    | "-" ~ !NUMBER ~ word_cont+
 }
 
 // ── EDN VALUE (any EDN atom or container) ─────────────────────────────────