adityamukho · Oct 20, 2026 · Oct 20, 2026
@@ -105,7 +105,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - `tests/snapshot_atomicity_test.rs` races a writer against a reader: a query must see each transaction whole, whether written with `execute` or a `WriteTransaction` (#554), and must read one state for all its patterns (#555), in memory and on file. It also runs the v2.x checkpoint race from #558, which never failed on v3: copy-on-write checkpoints do not overwrite pages a published reader uses.
 - **The `datalog_eval` fuzz guard counts partial bindings** (#597). The guard skipped inputs whose patterns join to more than 50,000 rows, counting only the rows after the last pattern. A `not-join` body still runs on the executor pattern by pattern in text order, so a 120-pattern body built hundreds of thousands of partial bindings before its last pattern matched nothing, and ran past the 60-second timeout. The guard now counts the bindings after every pattern. On the `v3` corpus (4,783 inputs) coverage drops by 7 of 9,719 edges. The input is added as a seed, as is the `btree_page` input of the cursor fix above.
 
-- **Parser vs grammar verifier (#643).** `src/query/datalog/parser/verifier.rs` checks, for every generated input, that the parser rejects what `tests/grammar/grammar.pest` rejects, and accepts what it accepts unless the error code is semantic (a listed code with a witness pest accepts). It runs every string of 1 to 3 characters over 102 characters (ASCII, `\t \n \r`, four non-ASCII stand-ins) through `parse_edn`, and every bracket-balanced sequence of up to `MINIGRAF_GRAMMAR_BOUND` (default 3) of 32 tokens in 15 command templates through `parse_datalog_command`, also with no separators up to 3 tokens. Deterministic; about 20 s by default and 3.5 min at bound 4 on a 16-thread machine. The drift it found is listed in `KNOWN_DRIFT` by cause (comments and non-ASCII letters #650; empty sections, ignored keywords, rule and call heads, a lone `:` or `-`, unterminated strings #651; a fact of more than four elements losing elements, #649, since fixed); each fix removes its entry. `grammar.pest` now says what the parser means: a fact is exactly `[e a v]` plus an optional valid-time map and a retracted fact exactly `[e a v]`; `not`, `not-join`, `or`, `or-join` (and `and` inside an `or`) are never rule names; literal keywords end at a word boundary (`:find?x` is one keyword).
+- **Parser vs grammar verifier (#643).** `src/query/datalog/parser/verifier.rs` checks, for every generated input, that the parser rejects what `tests/grammar/grammar.pest` rejects, and accepts what it accepts unless the error code is semantic (a listed code with a witness pest accepts). It runs every string of 1 to 3 characters over 102 characters (ASCII, `\t \n \r`, four non-ASCII stand-ins) through `parse_edn`, and every bracket-balanced sequence of up to `MINIGRAF_GRAMMAR_BOUND` (default 3) of 32 tokens in 15 command templates through `parse_datalog_command`, also with no separators up to 3 tokens. Deterministic; about 20 s by default and 3.5 min at bound 4 on a 16-thread machine. The drift it found is listed in `KNOWN_DRIFT` by cause (comments and non-ASCII letters #650, since documented: see below; empty sections, ignored keywords, rule and call heads, a lone `:` or `-`, unterminated strings #651; a fact of more than four elements losing elements, #649, since fixed); each fix removes its entry. `gr
8000
ammar.pest` now says what the parser means: a fact is exactly `[e a v]` plus an optional valid-time map and a retracted fact exactly `[e a v]`; `not`, `not-join`, `or`, `or-join` (and `and` inside an `or`) are never rule names; literal keywords end at a word boundary (`:find?x` is one keyword).
+- **The grammar documents comments and non-ASCII identifiers** (#650). The parser has always accepted both; `tests/grammar/grammar.pest` and `docs/GRAMMAR.md` now say so. A `;` starts a comment that runs to the end of the line and is skipped like whitespace. Symbols, variables, keywords and bind slots take Unicode letters and numerals (`:person/名前`, `?ñ`), as Rust's `char::is_alphabetic` and `char::is_numeric` define them; a symbol starts with a letter or `_`, and `-` followed by a numeral starts a number, not a symbol. Nothing the parser accepts or rejects changes.
 
 ### Documentation
 

@@ -129,7 +129,7 @@ cargo run < demos/demo_negation.txt
 
 3. **`src/query/datalog/`** — Datalog engine
    - `parser.rs`: EDN/Datalog parser — `transact`, `retract`, `query`, `rule`, `:as-of`, `:valid-at`, `not`, `not-join`; `resolve_pseudo_sources` names each per-fact pseudo-attribute's source (nearest earlier same-entity pattern, lexical scope) and its hidden metadata variable `?#<n>.<attr>`, else PRS-081 (#594)
-   - `parser/verifier.rs`: test-only (not wasm32) bounded-exhaustive check of the parser against `tests/grammar/grammar.pest` (#643): every 1–3 character string (`parse_edn`) and every balanced token filler up to `MINIGRAF_GRAMMAR_BOUND` (default 3) in 15 command templates (`parse_datalog_command`); pest rejects ⇒ parser rejects, pest accepts ⇒ parser accepts or raises a `SEMANTIC` code; today's drift is `KNOWN_DRIFT`, one entry per cause, removed by its fix (#650, #651; #649 fixed)
+   - `parser/verifier.rs`: test-only (not wasm32) bounded-exhaustive check of the parser against `tests/grammar/grammar.pest` (#643): every 1–3 character string (`parse_edn`) and every balanced token filler up to `MINIGRAF_GRAMMAR_BOUND` (default 3) in 15 command templates (`parse_datalog_command`); pest rejects ⇒ parser rejects, pest accepts ⇒ parser accepts or raises a `SEMANTIC` code; today's drift is `KNOWN_DRIFT`, one entry per cause, removed by its fix (#651; #649, #650 fixed)
    - `executor.rs`: Query executor — temporal filter (tx-time → net-assert → valid-time), not/not-join post-filter
    - `matcher.rs`: Pattern matching with variable unification; `edn_to_value`, `edn_to_entity_id`
    - `aggregate.rs`: `Agg` (one aggregate of one group) and `Running` (a window's running value), shared by the executor and the engine; exact sums (`i128` integers saturated once, fixed-point floats rounded once), `value_order` (one total order of values for window sorts, peers, `rank` and `min`/`max`), `named_vars` (a window's tie-break variables), `windows` (each window's value per row: partitions, sort, peers) (#547)
@@ -206,7 +206,7 @@ Auto-migrates v7 → v8 on open (spec §9, with a backup meta page). v1–v6 are
 
 ## Test Coverage
 
-**1661 tests** (1651 passing, 10 ignored; unit + integration + doc).
+**1662 tests** (1652 passing, 10 ignored; unit + integration + doc).
 See `docs/TEST_COVERAGE.md` for the full per-file breakdown.
 
 **Testing conventions** — see the Testing Conventions section below before writing any tests.

@@ -131,7 +131,7 @@ let r2 = pq.execute(&[("tx", BindValue::TxCount(2)), ("entity", BindValue::Entit
 
 ```bash
 cargo run          # interactive Datalog REPL
-cargo test         # run 1661 tests
+cargo test         # run 1662 tests
 cargo run < demos/demo_recursive.txt   # recursive rules demo
 ```
 

@@ -4,13 +4,13 @@
 
 This EBNF is the structural syntax of Minigraf's Datalog commands. It is generated from `tests/grammar/grammar.pest`, the grammar the parser is tested against: for every input up to a size bound, a test checks that the parser rejects what the grammar rejects, and accepts what it accepts unless a semantic rule rejects it. To change this page, edit the grammar and run `UPDATE_GRAMMAR_DOC=1 cargo test --test grammar_doc`.
 
-Between tokens the parser skips spaces, tabs, newlines, carriage returns and commas. Inside a rule marked `(* token *)` nothing is skipped, and the rules it uses are read the same way.
+Between tokens the parser skips spaces, tabs, newlines, carriage returns and commas. It also skips comments: `;` and the rest of its line. Inside a rule marked `(* token *)` nothing is skipped, and the rules it uses are read the same way.
 
 EBNF has no lookahead, so a pest lookahead appears as a comment: `(* not followed by x *)` means the next characters must not match `x`, and nothing is read. Literals have no escapes: `"\"` is one backslash and `'"'` one double quote.
 
 ```ebnf
 (* ── WORD-BOUNDARY HELPER ── *)
-word-cont            ::= letter | digit | "?" | "_" | "-" | "/"
+word-cont            ::= letter | numeral | "?" | "_" | "-" | "/"
 kw-find              ::= ":find" (* not followed by word-cont *)  (* token *)
 kw-where             ::= ":where" (* not followed by word-cont *)  (* token *)
 kw-with              ::= ":with" (* not followed by word-cont *)  (* token *)
@@ -38,12 +38,12 @@ str-char    ::= "\" ("n" | "t" | "r" | '"' | "\" | any-char)
               | (* not followed by '"' | "\" *) any-char
 string-lit  ::= '"' str-char* '"'  (* token *)
 uuid-lit    ::= "#uuid" string-lit
-bind-slot   ::= "$" (letter | digit | "_" | "-")+  (* token *)
+bind-slot   ::= "$" (letter | numeral | "_" | "-")+  (* token *)
 keyword     ::= ":" word-cont+  (* token *)
 variable    ::= "?" word-cont*  (* token *)
 op-sym      ::= "<=" | ">=" | "!=" | "<" | ">" | "=" | "+" | "*" | "/"  (* token *)
 plain-sym   ::= (* not followed by boolean-lit *) (* not followed by nil-lit *) (letter | "_") word-cont*
-              | "-" word-cont+  (* token *)
+              | "-" (* not followed by numeral *) word-cont+  (* token *)
 
 (* ── EDN VALUE (any EDN atom or container) ── *)
 edn-value  ::= uuid-lit | boolean-lit | nil-lit | float-lit | integer-lit | string-lit
@@ -121,7 +121,7 @@ rule-cmd    ::= "(" "rule" rule-vector ")"
 rule-vector ::= "[" rule-head where-clause* "]"
 rule-head   ::= "(" plain-sym variable* ")"
 
-(* letter = "a".."z" | "A".."Z"; digit = "0".."9"; any-char = any one character *)
+(* letter = a character Unicode calls Alphabetic; numeral = a character in Unicode's number categories (Nd, Nl, No), digits included; digit = "0".."9"; any-char = any one character *)
 ```
 
 Input that matches this grammar can still be rejected: the parser also enforces the rules in the Semantic Constraints section of the Datalog reference (variable safety, binding, aggregate and window compatibility, UUID and timestamp formats).
@@ -224,12 +224,8 @@ fn is_semantic(code: &str) -> bool {
 #[derive(Clone, Copy, PartialEq, Eq, Debug)]
 enum Cause {
     // Lexical.
-    /// `;` starts a line comment in the parser; the grammar has no comments.
-    Comment,
     /// A string with no closing quote reads to the end of the input.
     UnterminatedString,
-    /// A non-ASCII letter or digit in a symbol or keyword (the grammar is ASCII).
-    UnicodeLetter,
     /// `:` with no keyword character after it is the keyword `:`.
     BareColon,
     /// `-` with no word character after it is the symbol `-`.
@@ -260,11 +256,7 @@ enum Cause {
 impl Cause {
     fn level(self) -> Level {
         match self {
-            Cause::Comment
-            | Cause::UnterminatedString
-            | Cause::UnicodeLetter
-            | Cause::BareColon
-            | Cause::LoneMinus => Level::Lexical,
+            Cause::UnterminatedString | Cause::BareColon | Cause::LoneMinus => Level::Lexical,
             _ => Level::Structural,
         }
     }
@@ -278,21 +270,11 @@ struct Drift {
 }
 
 const KNOWN_DRIFT: &[Drift] = &[
-    Drift {
-        cause: Cause::Comment,
-        issue: 650,
-        min_k: 0,
-    },
     Drift {
         cause: Cause::UnterminatedString,
         issue: 651,
         min_k: 0,
     },
-    Drift {
-        cause: Cause::UnicodeLetter,
-        issue: 650,
-        min_k: 0,
-    },
     Drift {
         cause: Cause::BareColon,
         issue: 651,
@@ -373,14 +355,12 @@ fn lexical_cause(text: &str) -> Option<Cause> {
                     }
                 }
             }
-            ';' => return Some(Cause::Comment),
             ':' if !next.is_some_and(symbol_char) => return Some(Cause::BareColon),
             '-' if !next.is_some_and(symbol_char)
                 && !(i > 0 && (symbol_char(c[i - 1]) || matches!(c[i - 1], ':' | '#' | '$'))) =>
             {
                 return Some(Cause::LoneMinus);
             }
-            ch if !ch.is_ascii() && ch.is_alphanumeric() => return Some(Cause::UnicodeLetter),
             _ => {}
         }
         i += 1;
@@ -458,11 +438,14 @@ fn query_cause(items: &[EdnValue]) -> Option<Cause> {
 
 /// Whether pest's `plain_sym` reads all of `s`.
 fn is_plain_symbol(s: &str) -> bool {
-    let word = |c: char| c.is_ascii_alphanumeric() || matches!(c, '?' | '_' | '-' | '/');
     let mut chars = s.chars();
     match chars.next() {
-        Some(c) if c.is_ascii_alphabetic() || c == '_' => chars.all(word),
-        Some('-') => !chars.as_str().is_empty() && chars.all(word),
+        Some(c) if c.is_alphabetic() || c == '_' => chars.all(symbol_char),
+        Some('-') => {
+            !chars.as_str().is_empty()
+                && !chars.as_str().starts_with(char::is_numeric)
+                && chars.all(symbol_char)
+        }
         _ => false,
     }
 }

@@ -1,9 +1,13 @@
 // ── WHITESPACE (auto-skipped between every token) ─────────────────────────
 WHITESPACE = _{ " " | "\t" | "\n" | "\r" | "," }
 
+// Line comment: ";" to the end of the line, skipped like whitespace (#650).
+COMMENT = _{ ";" ~ (!"\n" ~ ANY)* }
+
 // ── WORD-BOUNDARY HELPER ──────────────────────────────────────────────────
-// Used in negative lookaheads to enforce keyword boundaries.
-word_cont = _{ ASCII_ALPHANUMERIC | "?" | "_" | "-" | "/" }
+// Used in negative lookaheads to enforce keyword boundaries. Letters and numerals are
+// Unicode's (Alphabetic; Nd, Nl, No), as the parser reads them (#650).
+word_cont = _{ ALPHABETIC | NUMBER | "?" | "_" | "-" | "/" }
 
 // Literal keywords: atomic, so that no whitespace is skipped before the
 // boundary check (":find?x" is one keyword, as the parser reads it).
@@ -52,7 +56,7 @@ string_lit = @{ "\"" ~ str_char* ~ "\"" }
 uuid_lit = { "#uuid" ~ string_lit }
 
 // Prepared-query bind slot: $name
-bind_slot = @{ "$" ~ (ASCII_ALPHANUMERIC | "_" | "-")+ }
+bind_slot = @{ "$" ~ (ALPHABETIC | NUMBER | "_" | "-")+ }
 
 // Keyword: starts with ":", followed by word constituents (incl. "?")
 keyword = @{ ":" ~ word_cont+ }
@@ -63,11 +67,12 @@ variable = @{ "?" ~ word_cont* }
 // Operator symbols used in expressions
 op_sym = @{ "<=" | ">=" | "!=" | "<" | ">" | "=" | "+" | "*" | "/" }
 
-// Plain symbol: letter or underscore start; or "-" followed by at least one word char.
+// Plain symbol: letter or underscore start; or "-" followed by at least one word char
+// that is not a numeral (after "-" the parser reads a numeral as a number).
 // Negative lookaheads prevent "true", "false", "nil" from matching as plain symbols.
 plain_sym = @{
-    !boolean_lit ~ !nil_lit ~ (ASCII_ALPHA | "_") ~ word_cont*
-    | "-" ~ word_cont+
+    !boolean_lit ~ !nil_lit ~ (ALPHABETIC | "_") ~ word_cont*
+    | "-" ~ !NUMBER ~ word_cont+
 }
 
 // ── EDN VALUE (any EDN atom or container) ─────────────────────────────────