8000 Lexer trivia with attachment; `Kex.AST.toSource` round-trip · Issue #136 · kexhq/kex · GitHub
[go: up one dir, main page]

Skip to content

Lexer trivia with attachment; Kex.AST.toSource round-trip #136

Open

Description

@akoskovacs

Part of #131. Blocked by #132, #135.

The lexer discards comments outright — Lexer::skipComment (src/lexer/lexer.cxx:313) advances to end-of-line and emits nothing. Doc-comments survive only because src/interpreter/stdlib/parser.cxx separately re-scans the raw source and associates # blocks with the line that follows them.

That is enough for docgen and not enough for anything that rewrites source. This stage makes the exposed tree lossless.

Not comment tokens

Emitting comment tokens would cascade into every token consumer — the parser, and through it the typechecker and backends — which is exactly the cascade the original design chose a source pre-scan to avoid. Instead:

Attachment, not lookup

A comment must be attached to the node it belongs to — leading this node, or trailing on the same line as it — rather than merely indexed by byte offset. Two consumers need this and neither can be served by a lookup table:

  • A formatter re-decides layout and moves nodes; a comment must move with the node it documents.
  • A linter binds # kex:disable-next-line to the node it precedes, not to a position.

Blank runs need counts, not just presence: gofmt and mix format preserve one blank line between declarations and collapse longer runs, which is only expressible if the model records how many there were.

Doc-comment extraction then stops being a separate line scanner and becomes a read of this table.

Kex.AST.toSource

An identity printer: given a parsed file, reproduce it byte for byte. Write it in Kex, not C++ — it is the strongest possible pin on lossl 6E88 essness, it is the seed of the eventual formatter, and it is an honest measure of whether the schema is good enough to build a compiler on.

Note it is not a formatter. toSource reproduces; a formatter discards the original layout on purpose and re-decides it via a layout algebra while preserving every comment. The formatter is out of scope here.

Verification

  • make test-all, both backends.
  • The round-trip run over every .kex file in examples/, spec/, and src/stdlib/ — several hundred real files, which is the only trivia test worth having. Any byte difference is a failure.

Activity

  1. changed the title [-]Lexer trivia with attachment; `Kex.Ast.toSource` round-trip[/-] [+]Lexer trivia with attachment; `Kex.AST.toSource` round-trip[/+] on Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0