8000 Expose the AST to Kex programs (`Kex.AST`) · Issue #131 · kexhq/kex · GitHub
[go: up one dir, main page]

Skip to content

Expose the AST to Kex programs (Kex.AST) #131

Open

Description

@akoskovacs

Kex programs cannot see the AST of Kex source in any complete or portable way. Today src/interpreter/stdlib/parser.cxx converts the C++ AST into a curated set of Kex values (Program/Node/Expression/TypeRef/PatternRef) with doc-comments and locations, declared in src/stdlib/parser.kex and consumed by tools/docgen/docgen.kex. Two properties limit it:

  • It is a hand-curated subset. Function bodies are omitted by design; bundle "demo" do ... end parses to a Call with the entire block missing; tey("greet", git: ..., tag: ...) loses its named arguments; Match keeps arm bodies but not their patterns.
  • It exists only in the tree-walk interpreter. Under -R it raises undef — there is no kex_intrinsic_parser.erl — and Tey runs on BEAM.

The consequence is recorded in docs/tey-limitations.md: Tey reads package.kex with a scanner that recognises declarations by their leading word, which can drift from what the compiler actually accepts, so a manifest Kex parses is one Tey may reject.

The inversion

The point of this work is not to improve a reporting view of the AST. It is to make src/stdlib/ast.kex the canonical AST schema, with the C++ compiler as its first producer rather than its owner. Everything below moves a piece of the compiler from "owns its own AST" toward "produces the schema", so that a Kex-native parser can later slot in behind the same types with no user-visible change.

Two decisions that make it iterative

  1. One converter, two sinks. The conversion is written once against a builder interface, with a ValueBuilder (interpreter ValuePtr) and a TermBuilder (src/beam/etf.hxx). Every fidelity increment then lands on the interpreter and BEAM simultaneously, and the two cannot drift.
  2. Trivia as attachment, not a token-stream change. The lexer records comments and blank runs in a side table; the converter attaches them to nodes. The token stream is untouched, so nothing cascades into the parser, typechecker, or backends.

Stages

Out of scope

Deliberately enabled but not built here:

  • Code formatter / pretty-printer in Kex. Note it is not toSource: toSource is an identity printer (the losslessness test), a formatter discards the original layout on purpose and re-decides it via a layout algebra while keeping every comment.
  • Linter in Kex. A syntactic linter needs only the CST; suppression comments (# kex:disable-next-line) are why trivia must be attached to nodes rather than looked up by offset — the same requirement the formatter has.
  • LSP refactors (rename, code actions, formatting). Stages 1 and 5 remove two of the three obstacles — precise spans, and lossless re-emission — but a correct rename also needs name resolution, which lives in src/semantic/ and is not exposed by this work. src/lsp/server.cxx is untouched throughout.
  • A resolved/semantic surface. The natural second surface after this one: what a semantic linter, LSP refactors, and a self-hosted typechecker all need. Strictly bigger, and should follow rather than be tangled in.
  • A Kex-native lexer and parser. This design is what that slots into.
  • A versioned on-disk AST artifact (.kexa). Stage 3 is a pipe, not a format. A bootstrap chain will eventually want a header; adding one is additive.

Activity

  1. changed the title [-]Expose the AST to Kex programs (`Kex.Ast`)[/-] [+]Expose the AST to Kex programs (`Kex.AST`)[/+] on Aug 19, 2026
  2. added this to the 0.4.0 milestone on Aug 20, 2026
  3. akoskovacs commented on Aug 21, 2026

    @akoskovacs
    CollaboratorAuthor

    Stage check against main (fc9a91d): #132, #133, #134, #135 and #137 are done and closed. #136 (lexer trivia with attachment, Kex.AST.toSource round-trip) is the only stage left — Kex.AST keeps doc-comments (doc : String?) but not ordinary comments or blank runs, so nothing can reproduce source yet. #169 (linter/autoformatter) depends on exactly that, so #136 is now its blocker too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0