Repository navigation
Description
Kex programs cannot see the AST of Kex source in any complete or portable way. Today src/interpreter/stdlib/parser.cxx converts the C++ AST into a curated set of Kex values (Program/Node/Expression/TypeRef/PatternRef) with doc-comments and locations, declared in src/stdlib/parser.kex and consumed by tools/docgen/docgen.kex. Two properties limit it:
- It is a hand-curated subset. Function bodies are omitted by design;
bundle "demo" do ... endparses to aCallwith the entire block missing;tey("greet", git: ..., tag: ...)loses its named arguments;Matchkeeps arm bodies but not their patterns. - It exists only in the tree-walk interpreter. Under
-Rit raisesundef— there is nokex_intrinsic_parser.erl— and Tey runs on BEAM.
The consequence is recorded in docs/tey-limitations.md: Tey reads package.kex with a scanner that recognises declarations by their leading word, which can drift from what the compiler actually accepts, so a manifest Kex parses is one Tey may reject.
The inversion
The point of this work is not to improve a reporting view of the AST. It is to make src/stdlib/ast.kex the canonical AST schema, with the C++ compiler as its first producer rather than its owner. Everything below moves a piece of the compiler from "owns its own AST" toward "produces the schema", so that a Kex-native parser can later slot in behind the same types with no user-visible change.
Two decisions that make it iterative
- One converter, two sinks. The conversion is written once against a builder interface, with a
ValueBuilder(interpreterValuePtr) and aTermBuilder(src/beam/etf.hxx). Every fidelity increment then lands on the interpreter and BEAM simultaneously, and the two cannot drift. - Trivia as attachment, not a token-stream change. The lexer records comments and blank runs in a side table; the converter attaches them to nodes. The token stream is untouched, so nothing cascades into the parser, typechecker, or backends.
Stages
- Add byte spans to
SourceLocationand populate them in the parser #132 — byte spans onSourceLocation - Refactor AST→value conversion onto one converter with two sinks #133 — one converter, two sinks
-
kex --emit-astandkex_intrinsic_ast.erl: AST access on BEAM #134 —kex --emit-astand the BEAM intrinsic - Full-fidelity AST schema: named args, call blocks, patterns, bodies #135 — full-fidelity schema
- Lexer trivia with attachment;
Kex.AST.toSourceround-trip #136 — lexer trivia andtoSource - Rename to
Kex.AST; migrate docgen and Tey's manifest reader #137 — rename toKex.AST, migrate consumers
Out of scope
Deliberately enabled but not built here:
- Code formatter / pretty-printer in Kex. Note it is not
toSource:toSourceis an identity printer (the losslessness test), a formatter discards the original layout on purpose and re-decides it via a layout algebra while keeping every comment. - Linter in Kex. A syntactic linter needs only the CST; suppression comments (
# kex:disable-next-line) are why trivia must be attached to nodes rather than looked up by offset — the same requirement the formatter has. - LSP refactors (rename, code actions, formatting). Stages 1 and 5 remove two of the three obstacles — precise spans, and lossless re-emission — but a correct rename also needs name resolution, which lives in
src/semantic/and is not exposed by this work.src/lsp/server.cxxis untouched throughout. - A resolved/semantic surface. The natural second surface after this one: what a semantic linter, LSP refactors, and a self-hosted typechecker all need. Strictly bigger, and should follow rather than be tangled in.
- A Kex-native lexer and parser. This design is what that slots into.
- A versioned on-disk AST artifact (
.kexa). Stage 3 is a pipe, not a format. A bootstrap chain will eventually want a header; adding one is additive.
Activity
[-]Expose the AST to Kex programs (`Kex.Ast`)[/-][+]Expose the AST to Kex programs (`Kex.AST`)[/+]akoskovacs commentedon Aug 21, 2026
Stage check against main (fc9a91d): #132, #133, #134, #135 and #137 are done and closed. #136 (lexer trivia with attachment,
Kex.AST.toSourceround-trip) is the only stage left —Kex.ASTkeeps doc-comments (doc : String?) but not ordinary comments or blank runs, so nothing can reproduce source yet. #169 (linter/autoformatter) depends on exactly that, so #136 is now its blocker too.