8000 BEAM: `with` (capability substitution) is not lowered · Issue #181 · kexhq/kex · GitHub
[go: up one dir, main page]

Skip to content

BEAM: with (capability substitution) is not lowered #181

Closed

Description

@akoskovacs

capability already compiles on BEAM — it lowers as the ordinary module it is, and a program that declares one and calls it runs correctly with -R. Only with is missing:

error: 'with Clock = ...' is not supported on the BEAM backend yet; capabilities
run on the interpreter. The capability itself compiles — only replacing it does not.

(That message replaced IR lower: unimplemented expr node N3kex3ast8WithExprE.)

This is a design decision, not a gap in typing, and the two candidates differ in more than effort.

Option A — process dictionary

with does erlang:put({kex_cap, Name}, Value) and restores on exit; every capability-qualified call site checks erlang:get before falling through to the real module. This mirrors what the walker does today (a per-process stack saved and restored around each resume()).

Two real problems:

  • No after. The IR has TryCatch but no finally, so an error escaping a with would leave the binding installed. Usually the process dies and takes its dictionary with it, but if a trying higher up in the same process catches the error, the binding leaks — a divergence from the walker, whose restore is a destructor and therefore unconditional.
  • Spawn inheritance is not free. The process dictionary is not inherited, so spawn would have to copy a snapshot explicitly to match spec/capabilities.kex, where a child spawned inside a region sees the replacement and one spawned outside does not.

It also puts a runtime check on every capability call.

Option B — thread it, as the design argues

The counter-proposal on #143 is that a capability is an implicit parameter, threaded statically. The requirement inference needed for that now exists: the compiler knows which functions require which capabilities and why. Lowering would pass the capability to exactly those functions.

This is the designed semantics, and it removes every problem above: exception safety is trivial because there is no state to restore, spawn inheritance is ordinary closure capture, and there is no per-call check. The cost is arity changes on capability-requiring functions and their call sites, and it would make the BEAM implementation structurally different from the walker's.

Recommendation

B, because A knowingly ships a restore hole and a hot-path check to imitate an implementation the design already rejected. But B is the larger change, and the walker would eventually want the same treatment so the two backends do not diverge in mechanism.

Until this lands, spec/capabilities.kex carries # kex: skip-beam.

Activity

  1. added this to the 0.4.0 milestone on Aug 22, 2026
  2. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    Static threading needs a precise call graph first, and today's is not

    Agreed on threading over the process dictionary. Implementing it turned up a prerequisite that has to land first, and it is 10000 worth recording because it is not obvious.

    The call graph and the requirement map are keyed by bare function name (m_callGraph[def.name], analyzer.cxx). Two same-named functions in different modules share one entry:

    module A do
      foul tick() -> Integer = Clock.now()   # requires Clock
    end
    module B do
      foul tick() -> Integer = 42            # requires nothing
    end
    

    For foulness this is safe, because the effect bit only ever widens — attributing A's foulness to B costs precision, never correctness. For threading it is not safe, because the parameter list is the ABI:

    • Threading by name would give both ticks the parameter, which is consistent but coarse — and the coarseness spreads, since common names (read, get, run) collide readily across a unit.
    • Threading only unambiguous names is worse: a function whose name happens to collide and genuinely requires a capability would receive no parameter, and its capability call would silently fall through to the real implementation inside a with. A test that reads as mocked would quietly exercise production — the exact failure this whole feature exists to remove.

    So the honest ordering is: give the analyzer per-definition identity (key by FunctionDef*, resolve call edges through the resolved-call targets that already exist for method calls), then thread.

    What landed meanwhile

    Two defects in the requirement inference, both found while sizing this:

    • Module-level functions recorded no requirements at all. The module branch of computeTransitiveEffects was a drifted copy of processFn that never set the collection target, so inference only ever worked for top-level functions. The duplicate is gone; module members now go through processFn.
    • Ambiguous names over-reported. Until the call graph is precise, a name defined more than once is dropped from the requirement map rather than guessed at, so hover cannot claim a requirement a function does not have.

    The rest of the plan, unchanged

    Plumbing is confirmed available: lowerModules/lowerProgramTiered already take optional semantic results and the Analyzer is in scope at the compile site, so requirements can be passed rather than recomputed — recomputing a second fixpoint in lowering is the divergence docs/ufcs-dispatch-plan.md warns about.

    Then: append a hidden parameter per required capability to locally-defined functions; pass the current value at local call sites; lower with to a binding of that value; and at a capability call site dispatch on it, falling back to the real module when it holds the default sentinel. One atom comparison per capability call, no process state, exception-safe, and spawn inherits by ordinary closure capture.

  3. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    Revised: one context parameter, not one per capability

    Better than what I proposed above, and it removes the blocker rather than working around it.

    Instead of a hidden parameter per required capability, every foul function defined in the unit takes one context — a map from capability name to implementation.

    %% with Clock = Impl do Body end
    Ctx2 = Ctx#{<<"Clock">> => Impl},   %% Body lowered against Ctx2
    
    %% Clock.now() inside a function holding Ctx
    case maps:get(<<"Clock">>, Ctx, undefined) of
        undefined -> 'Kex.Clock':now();
        Impl -> now(Impl)
    end

    Why this dissolves the precision problem. The per-capability version needed to know exactly which functions require which capabilities, which is why the bare-name call graph was fatal. With one uniform context the ABI rule is flat — every foul function takes it — so nothing about correctness depends on the call graph at all. Requirement inference drops back to advisory, for hover and target checking, which is all it should ever have been load-bearing for.

    Everything else follows:

    • No state, so no restore and no after hole — a context is a value.
    • spawn captures it in the closure, as does any block. That is why pure combinators never need it even when the block they run is foul: the block closed over Ctx where it was written, so map stays untouched.
    • One map lookup per capability call, with the real module as the miss case.

    Pure functions genuinely never need it, since a pure function cannot call a capability method — they are foul — and a foul block created in a foul function already carries the context.

    Cost. Every foul function gains one parameter whether it uses it or not: 108 foul against 938 let in the stdlib, so about a tenth of functions, and the pure core where the hot loops live is untouched.

    The precedent already exists

    Trait dictionaries thread hidden arguments through definitions and call sites today — def.dictionaryTraits and dictionaryArity when lowering a definition, fnTraitParams and makeTraitDictionary at call sites. A capability context is the same shape, so this is following an established path in lower.cxx rather than inventing one.

    Decision to take before implementing

    This changes the ABI of every foul function in the prelude, so prelude artifacts rebuild. Fine in-tree, and nothing sealed is stranded now that .kexo is dropped — but it does mean capability sets eventually belong in the KexI chunk, or an external caller will pass the wrong arity. Worth deciding whether to land the KexI field in the same change or accept that cross-module foul calls are unsupported until the module system is finished.

  4. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    Implementation attempt: what works, and the one thing that stops it

    Tried the context threading. Two pieces are done and correct; a third is bigger than it looks, so I backed the change out rather than leave a half-threaded ABI in the tree. Recording the findings so the next pass is quick.

    The gate works, and it matters

    Nothing is emitted unless the unit declares a capability (capabilityModules non-empty). Verified: a program with no capability compiles byte-identically, so the prelude and every existing spec are untouched. That bounds the blast radius to capability-using programs, which is spec/capabilities.kex and nothing else — and it means the prelude ABI churn I worried about above does not happen at all until the stdlib is converted.

    Definitions: done, and the insertion point is unambiguous

    The context is a trailing parameter after any trait dictionaries, with def.arity adjusted to match. Purity is uniform across an overload group (the UFCS plan requires it), so group[0]->isFoul decides for the whole group. currentContextVar is set for the body and restored after, so calls inside pick it up.

    Call sites: the blocker

    lower.cxx emits a local call from several branches, not one. I patched the fnTraitParams path — the same one the trait dictionaries use — and it was not enough:

    kex_beamcap1: undefined function stamp/0 in main/0
    

    main calls a zero-arg function through the zeroArgFns / thunk branch, which never reaches the site I patched. Constants, constructors, method calls, and the generated dispatchers are further paths, and each needs the same append to stay arity-consistent. Threading is therefore an enumeration problem across every local-call emission site, and missing one shows up as undef at runtime rather than a compile error.

    The suite will not catch mistakes here. No spec in the tree declares a capability except spec/capabilities.kex, which is # kex: skip-beam — so ordinary runs stay green while capability programs break. Whoever picks this up should drive it from capability programs run under -R, not from make test-all.

    One correctness point found along the way

    Functions defined inside a capability module must not take a context. They are the default implementation, and giving them one turns 'Clock.now'/0 into /1, which breaks the very fallback the dispatch relies on:

    kex_beamcap1: undefined function 'Clock.now'/0 in stamp/1
    

    Remaining work, in order

    1. Enumerate every local-call emission path and append the context on each (gated as above).
    2. Exclude capability-module members from taking one.
    3. Lower with to Ctx2 = Ctx#{Name => Impl} over the body.
    4. Lower a capability call to maps:get(Name, Ctx, undefined) with the real module as the miss case.
    5. Drop # kex: skip-beam from spec/capabilities.kex.
  5. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    Landed: with now lowers on BEAM, threaded as one context

    spec/capabilities.kex no longer carries # kex: skip-beam — it runs on both backends with identical output, and spec-beam went 258 → 259 matching.

    deep -> at 1000
    deep -> at 7          # `with` reaching through two calls that thread nothing
    deep -> at 1000
    nested inner: at 2
    nested outer: at 1
    spawned inside: at 7
    spawned outside: at 1000
    

    The process semantics came out free

    This is the part worth recording. The walker needed a per-process binding stack saved and restored around every resume(), plus an explicit snapshot at spawn, to stop one process changing what another process's code means. On BEAM none of that was written: the context is a value, so a spawned closure captures it and a process spawned outside the region simply never received it. The identical spawned inside: 7 / spawned outside: 1000 result falls out of ordinary closure capture.

    That is the concrete payoff of threading over the process dictionary, and it is why the after/restore hole never arose — there is nothing to restore.

    How it is built

    • One context parameter, last, after any trait dictionaries, on every foul function in a unit that declares a capability.
    • Gated: a unit declaring no capability emits byte-identical output, so the prelude and every existing spec are untouched. The prelude ABI churn discussed above does not happen at all until the stdlib is converted.
    • with lowers to maps:put(Name, Impl, Ctx) with the body lowered against the extended map.
    • A capability call becomes maps:get(Name, Ctx, undefined) with the capability's own body as the miss case.
    • A capability's own members are excluded from taking a context — they are the default implementation, and giving them one turns 'Clock.now'/0 into /1 and breaks the fallback.

    The call-site enumeration, resolved

    As predicted, this was the work. Local calls are emitted from many branches; they now all route through one localCall helper so the context is appended in a single place. The sites that mattered were the plain call path, the zero-arg/thunk paths, the module-member path, and the UFCS method path — each found by running a capability program under -R and reading the arity mismatch, since make test-all cannot see these.

    Remaining gap: nested capabilities

    FS2.File.read(p) lowers as a UFCS call on the result of FS2.File rather than as a module-qualified call, so the capability dispatch never sees it. Split into spec/capabilities_nested.kex, which keeps # kex: skip-beam.

    This matters for the stdlib conversion, since every realistic capability there is nested (FS.File, FS.Directory). Fixing it means making the outer call's candidate resolution recognise a dotted capability receiver.

  6. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    Correction: the nested-capability diagnosis above was wrong

    I said FS2.File.read(p) "lowers as a UFCS call on the result of FS2.File". The Core Erlang shows something more specific, and it changes what the fix is.

    A nested module becomes its own BEAM module, so the capability's read is 'Kex.FS2.File':read/1 — plain nested modules confirm this works:

    module Outer do
      module Inner do
        foul greet(name: String) -> String do return "hi " + name end
      end
    end
    IO.printLine(Outer.Inner.greet("x"))     # "hi x" on BOTH backends
    

    Change module Inner to capability Inner and it breaks:

    kex_nestcap: undefined function greet/2 in use/2
    

    The call reaches the member through the bare-name UFCS path and appends the capability context there, producing greet/2 against a greet/1. The exclusion that keeps a capability's own members at their declared arity is keyed on the enclosing module, which the bare-name path never consults.

    So the fix is not about candidate resolution. It is that "this name belongs to a capability's own implementation" has to be knowable from the bare name at a UFCS call site — and doing that naively is unsafe, because a same-named ordinary function would then be excluded too and lose the context its definition expects.

    Where this leaves things

    Top-level capabilities are landed and verified on both backends; spec/capabilities.kex runs under -R (spec-beam 258 → 259). Nested capabilities are in spec/capabilities_nested.kex, walker-only, with the corrected diagnosis in its header.

    This is the last thing standing between the current state and converting the stdlib, since FS.File and FS.Directory are both nested.

  7. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    Nested capabilities: the actual cause, third time lucky

    Tested the two hypotheses above. Both were wrong, and the real cause is one line of my own dispatch.

    Not the capability-member exclusion. Removing it — letting a capability's own members take a context like any other foul function — changes nothing: top-level still works, nested still fails identically. So the arity of the capability's members is not the problem.

    It is the miss-branch. The dispatch I emit is:

    case maps:get(<<"Outer.Inner">>, Ctx, undefined) of
        undefined -> <call the capability's own body>;   %% <-- emitted as a LOCAL call
        Impl      -> greet(Impl, ...)
    end

    That first branch goes through localCall, which always builds Call{"", name, ...}. For a top-level capability the implementation is a local function in the same emitted module, so that is right. For a nested capability it is not: a nested Kex module becomes its own BEAM module, so the body lives in 'Kex.Outer.Inner':greet/N and the call has to be module-qualified. Emitting it locally produces undefined function greet/2 in use/2.

    Plain nested modules work precisely because the ordinary path knows to emit the external call; my branch bypasses that path and hand-builds a local one.

    The fix

    The miss-branch must emit whatever the ordinary module-call path would have emitted, rather than assuming local. The awkward part is structural: the normal path consumes the argument expressions and returns, while the dispatch needs those same arguments twice (once per branch) — which is why I hand-built the call in the first place. Binding the arguments to variables up front and then asking the ordinary path to build a call over those variables is the shape that works.

    This is the last blocker for converting the stdlib, since FS.File and FS.Directory are both nested.

  8. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    Nested capabilities: the evidence, and why my earlier three diagnoses were wrong

    Read the emitted Core instead of guessing. The cause is a UFCS-versus-module-path precedence problem that capabilities expose but do not cause.

    Setup. A nested capability with an implementation:

    module Outer do
      capability Inner do
        foul greet(name: String) -> String do return "hi " + name end
      end
    end
    record Loud do end
    make Loud, implement: Outer.Inner do
      foul greet(name: String) -> String = "HI " + name
    end
    foul use(n: String) -> String = Outer.Inner.greet(n)
    

    What is emitted for use:

    'use'/2 = fun (N, _ir_Ctx3) ->
      let _ir_T4 = call 'erlang':'error'(<<"... Undefined function: Outer.Inner">>)
      in apply 'greet'/2(_ir_T4, N)

    The receiver Outer.Inner is evaluated as a value, fails, and greet is applied UFCS-style to the result. The capability dispatch never fires — instrumenting it shows it is not reached at all.

    Why the make block matters. Without any implementation, the dispatch does fire and emits a correct miss branch (call 'Kex.Outer.Inner':'greet'(...)); it fails only because the override branch references a local receiver dispatcher that does not exist when nothing implements the capability. Add an implementation and the behaviour changes completely: a make-block method named greet makes Outer.Inner.greet(n) resolve through the receiver/UFCS machinery instead of the module path.

    Why top-level capabilities are unaffected. Clock.now() has a bare UpperIdentifier receiver, which resolves as a module path before UFCS gets a chance. A dotted receiver does not, and a capability always has both a module member and a same-named make method — so this shape is guaranteed to collide.

    Corrections to what I posted earlier in this thread: it is not "UFCS on the result of FS2.File" in the sense I meant, it is not the capability-member arity exclusion (removing it changes nothing), and it is not the miss-branch being local (that branch is correct — the emitter already maps a dotted name to 'Kex.Outer.Inner').

    What the fix has to do

    Make a dotted module path win over UFCS receiver dispatch when the path names a module in the unit — or, narrower and safer, when it names a capability. This is a precedence question in lowerMethodCall, not something the capability dispatch can fix from inside, since it never runs.

    Worth noting the same precedence issue should be reachable without capabilities: a nested module function whose name collides with any make-block method in the unit. Only the capability case is covered by a spec today.

  9. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    Root cause is not capability-specific: filed as #182.

    The same failure reproduces with a plain nested module and no capability anywhere, as long as some make block defines a method with the matching name. The walker is right, BEAM lowers the dotted module path as a value. Capabilities merely guarantee the collision, since a capability always has both a module member and a same-named make method implementing it.

    So nested capabilities on BEAM are blocked on #182 rather than on anything in this design. spec/capabilities_nested.kex stays walker-only until that lands.

  10. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    Nested capabilities work on BEAM now — the blocker was #182, which is fixed.

    spec/capabilities_nested.kex no longer carries # kex: skip-beam, so every capability spec runs on both backends (spec-beam 259 -> 261):

    REAL:a.txt
    FAKE:a.txt      # `with FS2.File = FakeFiles { canned: "FAKE:" }`
    REAL:a.txt
    

    That clears the last thing standing between this and converting the stdlib, since FS.File and FS.Directory are both nested.

  11. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    The last BEAM blocker: cross-unit override dispatch

    Recording the design, since this is the only thing between the current state and converting the stdlib.

    The problem. The capability dispatch emits

    case maps:get(Name, Ctx, undefined) of
        undefined -> <the capability's own body>;
        Impl      -> method(Impl, Args...)      %% a LOCAL receiver dispatcher
    end

    The override branch needs a compile-time symbol. That is fine when the unit being compiled also implements the capability, and wrong otherwise — and "otherwise" is the normal case, because the call site lives in the stdlib while the implementation replacing it lives in the test. erlc then rejects a module over a branch that is never reached:

    kex_kex_repl_session: undefined function read/2 in main/0
    

    What it needs instead. A runtime receiver dispatch: given the replacement value and a method name, find the module that owns that method for that value's type and apply it. The walker does this naturally because its dispatch is dynamic.

    Why it is new infrastructure. The runtime has only two registries, kex_display_records and kex_display_variants, both about rendering — nothing maps a record tag to the module owning its methods. The closest existing hook is kex_io:register_display/2, which each compiled module already calls at startup with the record layouts it owns; it knows both the tag set and its own module atom, so extending it to record tag → module is a contained change rather than a new startup path.

    Sketch:

    1. register_display/2 (or a sibling) also records {Tag, Module} in a persistent term.
    2. A helper — kex_capability:dispatch(Impl, MethodBin, Args) — reads the tag, looks up the owning module, and erlang:applys. It should raise the same "Undefined method: M for T" message the rest of the runtime uses when the lookup fails, so the two backends stay identical.
    3. The override branch calls that helper instead of naming a local function.

    Caveat worth checking first: a make block need not live in the same module as the record it targets, so tag → module is an approximation. KexiMethodOwnership already exists in the KexI chunk and may carry the precise answer; if so, populate the registry from it rather than from the record layouts.

    Once this lands, spec/capabilities.kex and the stdlib conversion both work on BEAM, and Mock.* can go.

  12. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    Cross-unit override dispatch: landed. The remaining blocker is the gate.

    The override branch no longer names a local symbol. kex_io:dispatch_method/3 resolves the implementation at runtime from the value's tag, and each compiled unit reports which module owns its records' methods through a new register_method_owners/1, emitted beside the existing register_display/2 — so this reuses the startup path rather than adding one.

    Two details that mattered:

    • The context has to ride along. A capability's members are foul by construction, so their implementations take the context too; dispatching [Receiver | Args] alone gave undef until the context was appended, exactly as a direct call would.
    • The failure message routes through the same undefined_method helper the rest of the runtime uses, so the backends stay identical when a lookup fails.

    This fixed the four REPL/filesystem failures that the first stdlib attempt caused — ctest went back to 19/19 with FS.File still marked as a capability.

    What still blocks the stdlib conversion

    One spec fails with the conversion applied, and it is the ABI question from earlier in this thread rather than a new bug:

    spec/process_task.kex   (# kex: run-beam)   ->   Internal error: undef
    

    The prelude compiles as one unit. Once any stdlib module is a capability, every foul prelude function takes a context — but a consumer program that declares no capability of its own does not thread, so every cross-unit call into the prelude mismatches by one argument.

    The gate that made threading safe ("emit nothing unless this unit declares a capability") is precisely what breaks cross-unit consistency. Two ways out:

    1. Thread unconditionally. Every foul function takes a context whether or not any capability exists. Consistent by construction, and the measured cost is bounded — 108 foul against 938 let in the stdlib, with pure code untouched. It gives up the current "byte-identical when no capability is declared" property.
    2. Make lowering aware of imported capabilities, so a consumer threads whenever the prelude does. Keeps the property for programs that link nothing with capabilities, but lowering currently receives ExternalModules, not ImportedInterfaces, so the flag would have to be carried across.

    (1) is simpler and matches what this thread already agreed — one context on every foul function. (2) preserves a nicety that stops mattering the moment the stdlib is converted, since the prelude will then always have a capability.

    src/stdlib/fs.kex is reverted again; everything else here is kept and the tree is green.

  13. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    Tested unconditional threading. It needs one more thing, and I had the reason backwards.

    Tried option (1) — drop the gate, every foul function takes a context. It fails, and not for the reason I gave above.

    spec/process_task.kex  ->  Internal error: undef
    

    I said the problem was a consumer failing to thread while the prelude did. It is the mirror image: with threading unconditional the prelude's own foul functions become start/2, double/2 and so on, but a consumer has no way to know that an imported function is foul, so it still emits Task.start/1. Local calls agree; imported ones do not.

    let t = Task.start { double(21) }    # Task.start is now start/2 in the prelude
    

    So neither the gated nor the ungated form is complete on its own:

    • Gated is self-consistent within a unit and is what is committed — a unit declaring no capability emits exactly what it did before, and the suite is green.
    • Ungated is consistent in principle but needs imported foulness at the call site, so a consumer appends the context to a call into the prelude.

    The good news is that the information already exists: KexiExport.isFoul is in the chunk today, and ImportedFunction/ImportedModuleInterface carry it into semantic analysis. What is missing is that lowering receives ExternalModules, not the imported interfaces — so the call site cannot see it. Threading that through (a set of foul export keys, populated from KexI alongside exportArity) is the concrete next step, and it is the same shape as the isCapability plumbing that already landed.

    Order of work, then:

    1. Carry imported foulness into ExternalModules.
    2. Append the context on calls to imported foul functions.
    3. Drop the gate.
    4. Mark FS.File a capability and delete the Mock.FS machinery.

    Everything else from this session is committed and green: runtime override dispatch, the KexI capability flag, imported-capability registration, and the local-call routing.

  14. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    Step 1 of the remaining plan is in: imported foulness reaches lowering

    ExternalModules.foulExports now carries which imported exports are foul, populated from KexiExport.isFoul in kexi_registry.cxx beside exportArity, and merged in the override path in main.cxx. Inert on its own — nothing reads it yet — and the suite is green with it.

    What remains, precisely

    Step 2: append the context on calls to imported foul functions. This is the same enumeration the local calls needed, on the external side. The emission points are the externalModules->exportToBeamFn lookups in lower.cxx (currently four: around lines 1540, 1666, 2617, 2732), each building a callE(Atom, BeamFn, ...). They should route through one helper that appends currentContext() when foulExports holds the qualified key, mirroring localCall.

    The same caution applies as before: make test-all cannot see mistakes here. A missed site shows up as undef at runtime in a capability program, not as a compile error. Drive it from a capability program under -R, and from the prelude build, which exercises every shape at once.

    Step 3: drop the gate (threadsCapabilities() returns true). Verified failing today only because of step 2 — with the gate off, the prelude's own Task.start becomes start/2 while consumers still emit start/1.

    Step 4: mark FS.File a capability and delete the Mock.FS machinery. Everything else for this is already in place: the KexI capability flag, imported-capability registration with trait synthesis, runtime cross-unit override dispatch, and the nested-module fix from #182. The conversion itself is one keyword.

    Verified along the way

    With the conversion applied and cross-unit dispatch in place, ctest is 19/19 and the only failure is spec/process_task.kex, which is exactly the step-2 gap — a prelude call whose arity changed. So the sequence above is not speculative; each step has a failing test that proves it is needed and will show when it is done.

  15. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    Tested the combined change: gate off + imported-foul append together

    This is the key result, so recording it precisely.

    They must land together, and together they work for capabilities. With the gate dropped and the imported append enabled, both capability specs pass on BEAM. Each half alone fails:

    • Append without dropping the gate: a capability program emits Task.start/2 while the prelude, declaring no capability, still exports start/1. Both capability specs die with undef (spec-beam 261 → 259).
    • Dropping the gate without the append: the prelude's own foul functions become start/2 while consumers still emit start/1. spec/process_task.kex dies with undef.

    What the combined change leaves. spec-beam goes to 258/3, and the three are not capability specs — they are call shapes that universal threading has not covered yet:

    Spec Failure Shape
    currying.kex badarg a function REFERENCE to a foul function — the fun's arity no longer matches the function it names
    crossfile_using.kex badarg same shape, across files
    optionparser_commands.kex undef a further call path, arity mismatch

    The two badargs are the interesting ones and probably one fix: ~foulFn (and any fun F/N reference) must account for the context, either by binding it — fun(A1..An) -> f(A1..An, Ctx) end — or by carrying the widened arity through the reference machinery, including Fun.arity bookkeeping in the runtime.

    Two arity bugs found and fixed along the way

    Both were mine, and both were the same mistake: recomputing a call's arity as args.size() where an explicit value was authoritative — ar at the external-call sites, resolved->second.backendArity on the resolved path. They do not always agree, and getting it wrong silently changed calls that had nothing to do with capabilities.

    State

    Reverted to the gated form, which is green (spec-beam 261/0). foulExports and the externalCallExpr routing stay, so enabling the pair is a two-line flip. The next piece of work is the function-reference shape, and it has a failing spec to prove it: spec/currying.kex under -R.

  16. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    A note on approach, having done it the slow way

    I found the call sites by trial: apply a capability, run under -R, read the arity mismatch, patch that site, repeat. That found the plain-call path, the zero-arg/thunk paths, the module-member path, the UFCS path, the sibling-module callE("", ...) paths, the resolved-call path, and finally the function-reference shape. Each round costs a full rebuild plus a suite run, and a missed site is invisible until a capability program runs — make test-all cannot see it.

    The systematic alternative is to centralize at callE. Every emitted call is built there, so if the append happened in that one place — driven by a table of known-foul targets rather than by whichever branch is being edited — every path would be covered by construction, including ones nobody has enumerated.

    What it needs:

    • A reverse table from backend identity to foulness. callE receives a module ATOM and a backend function name, while foulFns is keyed by local source name and foulExports by Short.name. Building {atom, beamFn} -> foul once from ExternalModules plus the local definitions closes that gap.
    • Runtime BIFs must be excluded, which falls out naturally: erlang:, maps: and kex_io: are not in either table.

    It does not cover function references. fun F/N is a different node, not a callE, which is exactly why currying.kex and crossfile_using.kex still fail with badarg after the call sites agree. That shape needs its own answer — binding the context into the reference (fun(A1..An) -> f(A1..An, Ctx) end) is the obvious one, and it also has to keep the runtime's Fun.arity bookkeeping honest, since kex_intrinsic_fun reads it.

    So the remaining work is two pieces, not N: centralize the call append, then handle references. That is a much smaller statement than the site-by-site list above, and it is what I would start from rather than continuing the trial loop.

  17. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    Centralizing worked, and the last failure is now exact

    Tried the centralized form described above: the append moved into callE, driven by a {atom, beamFn} -> foul table built once from ExternalModules, with the gate off.

    It reproduces the per-site result exactly — same three failing specs, no more, no fewer. That is the useful confirmation: the call-site problem is solved, one append covers every emission path, and the per-site enumeration was unnecessary. Runtime BIFs exclude themselves by not being in the table.

    The remaining defect, precisely

    spec/currying.kex:64:

    ["a", "b"].each(~IO.printLine)     # badarg
    

    The prelude now exports three arities for that one function:

    [{'IO.printLine',1}, {'IO.printLine',2}, {'IO.printLine',3}]
    

    The extra ones are the context-widened variants. The curry/reference path derives the lambda's parameter count from the callee's backend arity, so ~IO.printLine builds a lambda one parameter too wide; each then applies it with a single argument and Erlang raises badarg.

    The body is fine — it goes through buildCall, which appends the context correctly. Only the parameter count is wrong.

    Fix: the reference and curry paths must use the source parameter count, never the backend arity, because the context is added during lowering and is not a source parameter. exportParamNames already carries the source names, so it is likely a matter of ensuring that is what the arity is taken from on every branch of that path — including the fallback when it is missing, which is where the widened exportArity can leak in.

    crossfile_using.kex fails identically (badarg, same shape across files). optionparser_commands.kex fails with undef and has not been diagnosed.

    State

    Reverted to the gated form; the tree is green (spec-beam 261/0) with foulExports and the routing kept. Enabling the pair is a two-line change once the reference arity is fixed, and spec/currying.kex under -R is the proving test.

  18. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    The blocker is not a missed call site — the context collides with trait dictionaries

    Instrumented instead of guessing, and the previous explanation (curry taking a widened arity) is wrong: the probe shows the curry arity is 1, which is correct, and the emitted Core is correct too:

    F = fun (_ir_T0) -> call 'Kex.IO':'printLine'(_ir_T0, call 'maps':'new'()) end,
    apply F(<<"direct">>)

    The badarg is raised inside printLine/2, and that arity is not what I assumed:

    'printLine'/2 =
      fun (_ir_Fwd2469, _ir_Fwd2470) ->
        let <_ir_FwdDict2473> = ...        %% a trait DICTIONARY wrapper

    printLine(msg: Showable) emits printLine/1 plus a dictionary variant printLine/2. Threading makes the genuine target printLine/3. Appending the context produced /2, which lands on the dictionary wrapper and hands it the context map where a Showable dictionary belongs.

    Why this matters beyond one function

    A positional context can collide with an existing arity of the same name. Trait dictionaries already occupy trailing positions, so "append one more argument" is not a safe ABI extension on its own — the arity alone no longer identifies the target. This is not a missed call site and cannot be fixed by finding more of them.

    Whatever lands has to make the context unambiguous. Options, roughly in order of how much they disturb what exists:

    1. Mangle the threaded name rather than widening the arity — printLine and printLine$ctx never collide, and dictionaries keep their slots.
    2. Resolve the exact target arity at the call site (source arity + dictionary count + 1) rather than appending blindly, which means the caller must also know the dictionary count — coupling the two mechanisms.
    3. Carry the context inside an existing parameter, e.g. extending the dictionary tuple, so no arity changes at all.

    (1) looks cleanest: it keeps dictionaries untouched and makes "threaded" visible in the symbol, which also makes a mismatch a link error rather than a silent misinterpretation of an argument.

    Where this leaves things

    The gated form is committed and green (spec-beam 261/0). The three specs that fail with threading on — currying, crossfile_using, optionparser_commands — are all this same collision, not three separate gaps, which is why fixing call sites never converged.

    Also worth noting for whoever picks it up: each { |x| IO.printLine(x) } works while each(~IO.printLine) does not, because the direct call resolves the dictionary properly and the reference does not.

  19. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    BEAM with is done, and the stdlib conversion works

    Threading is now universal — every foul function takes the context, in every unit — and the whole suite is green with it: ctest 19/19, spec 277/0, spec-beam 262/0, prelude 462/0 and 453/0, stdlib 59/0 on both backends.

    FS.File is a real capability in src/stdlib/fs.kex, replaceable from user code on both backends with no global mock state. spec/capability_stdlib_fs.kex covers it.

    Confirming option C — the "collision" was an ordering bug

    My earlier framing (positional context fundamentally collides with trait-dictionary arities) was wrong. The wrapper was:

    'printLine'/2 = fun (Fwd1, Fwd2) -> ... 'Kex.IO':'printLine'(Fwd1, Fwd2, FwdDict)

    makeDictionaryWrapper derives its own parameter count from arity - dictionaryTraits.size(), which swallowed the context slot — so it forwarded (msg, Ctx) and appended the dictionary last, while the definition emits (msg, Dict, Ctx). The real function then called showValue on the context map. FunDef::hasCapabilityContext now tells the wrapper the slot exists; it takes the context as its own trailing parameter and forwards it after the dictionaries.

    That one fix took spec-beam from 258/3 to 260/1 — currying and crossfile_using both.

    The other two gaps it uncovered

    • Receiver functions carried no foulness. cli.run(...) resolves through ExternalModules.receiverFunctions, a separate table from exportToBeamFn, so receiver calls never got the context. ReceiverFunction::isFoul fixes it — that was the last failing spec.
    • Capability members were excluded from threading. Keeping them at their declared arity made them disagree with every threaded caller. They now thread like anything else, and the dispatch passes the context through.

    Still open, and why

    Converting prelude capabilities (ENV, IO, …) needs one more dispatch site. FS is opt-in, so using FS merges its source and its calls resolve through moduleFunctions, where the dispatch lives. A prelude module reaches a consumer only through the external tables, and ENV.get(...) lowers through a third path — the qualified FunctionCall route — which has no dispatch. The symptom is the dangerous one: with ENV = FakeEnv { … } compiles and runs, and silently uses the real environment.

    The three dispatch sites should be collapsed into one before adding a fourth. The callE centralisation worked for the append; the dispatch is harder because it replaces the call rather than augmenting it, but the same argument applies — enumerating sites is how this stayed broken.

    Deleting Mock.* waits on that, since Mock.ENV, Mock.IO and Mock.Http all cover prelude modules.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0