You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Lexer trivia with attachment; Kex.AST.toSource round-trip #136
The lexer discards comments outright — Lexer::skipComment (src/lexer/lexer.cxx:313) advances to end-of-line and emits nothing. Doc-comments survive only because src/interpreter/stdlib/parser.cxx separately re-scans the raw source and associates # blocks with the line that follows them.
That is enough for docgen and not enough for anything that rewrites source. This stage makes the exposed tree lossless.
Not comment tokens
Emitting comment tokens would cascade into every token consumer — the parser, and through it the typechecker and backends — which is exactly the cascade the original design chose a source pre-scan to avoid. Instead:
The lexer records a side table of {kind, span, text} for comments and blank runs, using the byte offsets it already computes.
A comment must be attached to the node it belongs to — leading this node, or trailing on the same line as it — rather than merely indexed by byte offset. Two consumers need this and neither can be served by a lookup table:
A formatter re-decides layout and moves nodes; a comment must move with the node it documents.
A linter binds # kex:disable-next-line to the node it precedes, not to a position.
Blank runs need counts, not just presence: gofmt and mix format preserve one blank line between declarations and collapse longer runs, which is only expressible if the model records how many there were.
Doc-comment extraction then stops being a separate line scanner and becomes a read of this table.
Kex.AST.toSource
An identity printer: given a parsed file, reproduce it byte for byte. Write it in Kex, not C++ — it is the strongest possible pin on lossl
6E88
essness, it is the seed of the eventual formatter, and it is an honest measure of whether the schema is good enough to build a compiler on.
Note it is not a formatter. toSource reproduces; a formatter discards the original layout on purpose and re-decides it via a layout algebra while preserving every comment. The formatter is out of scope here.
Verification
make test-all, both backends.
The round-trip run over every .kex file in examples/, spec/, and src/stdlib/ — several hundred real files, which is the only trivia test worth having. Any byte difference is a failure.
changed the title [-]Lexer trivia with attachment; `Kex.Ast.toSource` round-trip[/-][+]Lexer trivia with attachment; `Kex.AST.toSource` round-trip[/+]on Aug 19, 2026
Activity
[-]Lexer trivia with attachment; `Kex.Ast.toSource` round-trip[/-][+]Lexer trivia with attachment; `Kex.AST.toSource` round-trip[/+]