Repository navigation
Grammar: document or drop line comments and non-ASCII identifiers #650
Description
Summary
The parser accepts two things the grammar (tests/grammar/grammar.pest, and so the EBNF) does not describe:
- Line comments.
;to the end of the line is skipped (added in 19d0991).1;parses as1. - Non-ASCII letters and digits in symbols and keywords: the tokenizer uses
char::is_alphanumeric, so:person/名前,?ñandF٣are accepted. The grammar usesASCII_ALPHANUMERIC.
Both are useful and likely used, so this is a documentation decision first: either document them in the grammar (add COMMENT to grammar.pest, widen word_cont and plain_sym), or restrict the parser.
Found by the grammar verifier (#643): the Comment and UnicodeLetter entries of KNOWN_DRIFT in src/query/datalog/parser/verifier.rs. The fix removes them.
Activity
Merge pull request #658 from project-minigraf/docs/650-grammar-commen…
adityamukho commentedon Oct 20, 2026
Fixed on v3 by #658: the grammar now documents line comments and Unicode identifiers.
Merge origin/v3 into fix/651-parser-tightening (#650 grammar; KNOWN_D…
docs(grammar): document line comments and Unicode identifiers (#650) (#…