8000 Inferred capabilities and scoped overrides (replace global `Mock` state) · Issue #143 · kexhq/kex · GitHub
[go: up one dir, main page]

Skip to content

Inferred capabilities and scoped overrides (replace global Mock state) #143

Closed

Description

@akoskovacs

Successor to #130, and a sibling of #142. Where #142 makes the purity BIT sound, this is about making effects nameable and substitutable — replacing the global Mock machinery with something scoped and typed.

The problem it solves

Testing anything effectful today means swapping global interpreter state. spec/fs.spec.kex:

describe("File — read and write (mocked)") do
  before { Mock.FS.clear() }
  after  { Mock.FS.clear() }

  it("read returns the mocked content") do
    Mock.FS.File("greet.txt", "hello")
    assert(FS.File.read("greet.txt").or("") == "hello")
  end

Mock.FS.File(...) writes into a global table (Kex.Intrinsic.FS.file → src/interpreter/stdlib/*.cxx) that FS.File.read consults. Consequences: every block needs before/after clearing, a forgotten clear leaks into the next test, and specs cannot run in parallel. Mock.FS, Mock.Http, Mock.ENV, Mock.System and Mock.IO in src/stdlib/http.kex all work this way.

The underlying gap: there is no way to say "this function needs a filesystem" and hand it a different one.

The explicit form already works — and shows the cost

No language changes are needed to express a capability. This runs on main today:

trait FileStore do
  foul readFile : String -> String?
  foul writeFile : String -> String -> Bool
end

record FakeFS do
  files : Map<String, String>
end

make FakeFS, implement: FileStore do
  foul readFile(path: String) -> String? = @files.get(path)
  foul writeFile(path: String, body: String) -> Bool = true
end

foul loadGreeting(fs: FileStore, path: String) -> String do
  return fs.readFile(path).or("(missing)")
end

Traits already carry foul on required methods, and trait-typed parameters already dispatch. The cost is threading: every function between main and the filesystem grows a parameter. That is why this pattern has not been adopted, and it is the thing the proposal below removes.

Proposal: inferred capabilities + lexically scoped override

1. Infer the capability set; never write it. Widen isFoul from bool to a small set of atoms (IO, FS, Net, Process, Env), inferred from which intrinsics a function reaches. Effects only union upward, so the existing fixpoint in computeTransitiveEffects already has the right shape — bool OR becomes set union. The set is displayed (LSP hover, generated docs), not annotated, so signatures do not grow.

2. Override it in a scope.

# nothing annotated — inferred `foul(FS)`
let loadGreeting(path: String) -> String = FS.File.read(path).or("(missing)")

it("reads the mocked content") do
  with FS = FakeFS { files: { "greet.txt": "hello" } } do
    assert(loadGreeting("greet.txt") == "hello")
  end
end

No signature change, no threading, no ABI change. The override dies with the block, so the before/after clearing disappears and specs become parallel-safe.

Rejected alternative: inferred implicit parameter

The compiler could instead add a hidden argument per capability, defaulted to the real implementation (Scala implicits / modular implicits). This changes the ABI of every foul function and forces BEAM codegen to thread the extra argument through every call. Much more invasive for the same testing benefit.

The open question — and the design risk

How does a scope interact with the process model?

If loadGreeting runs inside a spawn, does the spawned process see the override? Elixir hit exactly this with Mox and had to add explicit allow/3 so one process can lend a mock to another — widely regarded as the painful part of that library. Decisions needed before any implementation:

  • Is a with scope inherited by processes spawned inside it?
  • What happens when a spawned process outlives the scope?
  • On BEAM this is presumably a process-dictionary entry, which makes inheritance a concrete choice rather than a theoretical one.

Everything else is tractable: inferring the set, rendering it, and scoping the override in the tree-walker.

Dependencies and scope notes

  • Depends on step 1. There is nothing to override until effects are named, so the bool→set widening lands first. It is independently useful: it delivers the target capability declarations docs/purity.md already promises (target: wasm, capabilities: [IO]) and never implemented.
  • Independent of Effect polymorphism: a foul block escapes through a pure higher-order function #142. That closes the higher-order soundness hole; this one names and substitutes effects. Neither blocks the other.
  • compiled do is unaffected — compiled-block code is already required to be pure (src/compiled/expand.cxx:622) and enforced as such.
  • Inferred capabilities carry the usual interface-instability property: changing a body can change what a function requires. It only bites where a pure caller exists, which the checker already reports.

Not proposed

A general effect algebra with effect variables threaded through signatures. Kex's foul is deliberately closer to Haskell's single IO bit than to Koka's effect rows, and this proposal keeps that: effects stay inferred and unwritten. The goal is substitutability for testing and target capabilities, not effect polymorphism.

Design constraint: do not regress the ergonomic floor

The current Mock API's real virtue is that it asks nothing of the user. Stubbing a file read is one line:

Mock.FS.File("greet.txt", "hello")

No interface to declare, no fake type to write, no injection to wire. That is genuinely better than the equivalent in most languages, where testing a filesystem read means defining an interface, a stub implementation, and a DI seam first.

So the capability design must not require a user to write a FileStore trait and a FakeFS record in order to stub one read. If it does, it is a regression for the common case however much cleaner it is in principle. Concretely:

  • The stdlib ships the fakes. Mock.FS, Mock.Http, Mock.ENV, Mock.System and Mock.IO stay as the authoring API; with merely scopes them.
  • User-defined capabilities (the trait FileStore form above) are for a user's OWN effectful subsystems, never a prerequisite for testing the stdlib's.

Cheapest useful increment (does not need capabilities at all)

Most of the value here is available without naming effects or building any capability machinery: keep the Mock API exactly as it is, and scope its state to the enclosing it block automatically.

it("read returns the mocked content") do
  Mock.FS.File("greet.txt", "hello")        # unchanged
  assert(FS.File.read("greet.txt").or("") == "hello")
end                                          # state discarded here

That alone deletes every before { Mock.FS.clear() } / after { Mock.FS.clear() } pair 10BC8 in spec/, removes the forgotten-clear leak, and makes specs parallel-safe — with zero new user-facing concepts and no change to any signature. before/after hooks are real (src/interpreter/stdlib/test.cxx:87), so the test runner already has the block boundary to hang this on.

Worth doing this first and independently. It settles the process-model question in a much smaller arena (does mock state follow a spawn inside an it?), and if the full capability system never happens, the specs still get the win.

Activity

  1. akoskovacs commented on Aug 19, 2026

    @akoskovacs
    CollaboratorAuthor

    Filed #144: the test-only restriction Mock was intended to have was never implemented, so Mock.* is callable from any program — prelude.kex has using HTTP and module Mock lives in http.kex, so it is auto-imported everywhere. Verified hijacks of FS reads, ENV.get, System.OS and Http.get from a plain main block.

    That makes the scoping proposed here a security boundary, not only an ergonomic one: a lexically scoped override cannot outlive its block or reach a caller. #144's recommended first fix (gate the mock intrinsics on test mode) is orthogonal and shippable independently.

  2. akoskovacs commented on Aug 21, 2026

    @akoskovacs
    CollaboratorAuthor

    Counter-proposal: capabilities as implicit parameters, this-style

    Position in one line: a capability is a module that declares itself one, reaches code as an implicit parameter (static, this-style, never written in a signature), is substituted by a lexically scoped with, and is checked against the compilation target through the same inferred requirement. Not dynamic scope, not a written parameter, and not a widening of foul.

    Both stated reasons for rejecting the implicit-parameter alternative are survivable

    The ABI objection. .kexo is dropped, so there is no prebuilt binary a consumer is stuck with. But the objection is not simply void, and an earlier version of this comment overstated that — consumers compile against a dependency's KexI chunk, not its source (docs/borrowed-ideas.md:660-665; that is why cross-module specialization already works). A module interface boundary exists; it just is not .kexo.

    So the corrected position: a function's capability set is interface data, and changing a body can change it, which forces dependents to recompile. That is recompilation coupling, not an ABI break, and it is survivable precisely because .kexo is dropped — Tey always holds the source (git-only, compiled locally, docs/plan_tey_pm.md), so recompiling is always available. With sealed binary artifacts it would not have been.

    The codegen objection. Threading one argument through statically resolved call sites is what lowering already does (src/ir/lower.cxx:2211, :2270). The scoped-override design should be priced against that: a per-process capability stack, push/pop with exception safety, snapshot copying at every spawn, and a process-dictionary read on every capability call. The original comparison counted compiler work and not runtime work.

    The design

    A capability is an implicit parameter in exactly the sense this already is inside a make block: a real value, statically passed, type-checked, defaulted, never written in a signature. A function whose body reaches a capability module acquires one, and the compiler threads it.

    # nothing annotated, nothing threaded by hand
    let loadGreeting(path: String) -> String = FS.File.read(path).or("(missing)")
    
    it("reads the mocked content") do
      with FS.File = FakeFiles { files: { "greet.txt": "hello" } } do
        assert(loadGreeting("greet.txt") == "hello")
      end
    end
    

    The implicit argument defaults to the real module, so every call outside a with — all production code — compiles to the direct static call emitted today.

    Syntax: capability sits where module sits

    module FS do
      capability File do
        foul read(path) = Kex.Intrinsic.File.read(path)
        foul write(path, content) = Kex.Intrinsic.File.write(path, content)
      end
    
      module Path do              # untouched — pure, direct, no implicit parameter
        let join(a, b) = FS.Path.joinAll([a, b])
      end
    end
    

    FS itself cannot be the capability. src/stdlib/fs.kex splits into FS.File (all foul, all intrinsics), FS.Directory (same), and FS.Path (all let — join, dirname, basename, normalize). If FS were the capability, FS.Path.join would make its caller acquire an implicit parameter for pure string work with no implementation to substitute. Mocking dirname is nonsense. The capability boundary is the innermost module that owns effectful methods.

    capability declares interface and default implementation in one: the public signatures become a trait, so make FakeFiles, implement: FS.File works with existing machinery, and the bodies become the default the implicit parameter carries. That is why it is a one-keyword edit to the stdlib rather than a rewrite.

    The declaration lives on the capability, not in the compiler

    This issue proposes inferring capabilities "from which intrinsics a function reaches", which puts a hardcoded intrinsic→capability table in the analyzer. Better: the module that is the capability declares itself one, and propagation is then the existing transitive fixpoint (computeTransitiveEffects, src/semantic/analyzer.cxx:671). The compiler stops knowing anything about filesystems.

    • User-extensible for free. A user's PaymentGateway declares itself a capability and gets with, target checking, and acquisition chains with no compiler change.
    • Nothing is declared at a use site or in any signature.
    • A requires form remains as an escape hatch for what the call graph cannot see: FFI, and intrinsics not routed through a capability module.

    Why implicit rather than an ordinary parameter

    Writing the dependency as a normal parameter (fs: FileStore) turns every call site into fs.readFile(path) and loses FS.File.read(path).

    That is not only an API regression. A mock's value is that the test is evidence about production; if injecting a fake requires rewriting the call site, the code under test is not the code that ships. Every mechanism that makes the dependency reachable from the signature has this property. An implicit parameter is the only static option that leaves the call site byte-identical.

    Processes

    The rule is one line: a capability is a value, so spawn captures it like any other free variable (src/interpreter/evaluator.cxx:2670 already hands the child the parent's environment).

    with FS.File = FakeFiles { … } do
      let p = spawn do
        loadGreeting("greet.txt")     # sees the fake — captured at spawn
      end
    end                                # p may outlive this block; it holds the value
    

    Three consequences fall out rather than needing rules:

    • A child outliving the with keeps the fake. That is a closure outliving its defining scope — ordinary, and not the hazard it would be under dynamic scope.
    • Capabilities do not travel by message. A receiving process uses what it captured at spawn. One process, one world.
    • A process spawned outside any with never sees it. That is the supervisor case, and the answer is that the test owns the spawn — no allow/3.

    This is the decisive advantage over the scoped-override design: the spawn question is not answered, it stops existing. No inheritance rule, no snapshot semantics, no decision about a child outliving a scope.

    Process.run is a capability; spawn/receive are not

    These must be split, and the split is a cost decision as much as a semantic one.

    • Process.run — starting a child OS process. Genuinely absent under wasm, and genuinely substitutable (canned exit codes, stdout, stderr). A capability.
    • spawn / receive — the process model itself. Present everywhere Kex runs, and there is nothing to substitute: a fake scheduler is not a thing anyone wants. Not a capability.

    If the process model were a capability, every server loop would acquire the implicit parameter — and on BEAM that is not the cold effectful shell, it is the core of how programs are structured. Splitting keeps the threading cost off the hot path entirely while still catching the one case that actually diverges by target.

    Stateful fakes are processes, and this already runs

    No global mutable state means a fake that records calls must be backed by a process. That composes today, with no new machinery — verified on the walker:

    record SpyFiles do
      recorder : Process
    end
    
    make SpyFiles, implement: FileStore do
      foul read(path: String) -> String do
        @recorder.send(Read(path))
        return "fake:" + path
      end
    end
    

    Output: recorded: [a.txt, b.txt]. That is call verification, which Mock.* cannot do at all — it has canned responses and no way to assert what was called.

    Interaction with FileHandle<R, W> typestate

    docs/typestate-file-plan.md is implemented, and it changes the granularity story here — favourably.

    Typestate and capabilities answer different questions: typestate is what may I do with this resource I hold (per-value); a capability is which implementation backs the resource, and does this target have one (per-program). So FS.File stays a single capability and read/write safety rides on the handle it returns. There is no need to split FS.File into read/write capabilities — that granularity already exists, and #139 (Readable/Writable as named traits) is a separate axis with no bearing here: it is about letting non-FileHandle types implement the vocabulary, not about permission.

    It also makes the capability interface more precise, because the signature carries the typestate:

    capability File do
      foul open(path: FS.FilePath, mode: Read) -> Result<FileHandle<CanRead, CannotWrite>, IOError>
    end
    

    A fake physically cannot hand back a writable handle where a read-only one is due.

    The same mechanism solves backend divergence

    docs/compilation.md:29-46 already specifies capability inference for target support and marks it unimplemented. That is this machinery: with provides a capability for a lexical region, a target provides it for the whole program.

    The one remaining # kex: skip-wasm in the tree is the demonstration case. spec/process_streams.kex is skipped because Process.run has no child processes under wasm. Note this is Process.run specifically, not spawn/receive (see the split above) — and it is both axes at once:

    • Absent on a target — wasm does not provide Process, so it is a compile error rather than today's runtime string "process execution is unavailable in wasm".
    • Substitutable — with Process = FakeProcess { … } returning canned exit codes and stdout is a test that cannot be written today at all.

    The spec then stops needing a comment that Makefile:278 greps for: it does not compile for wasm, which is the truth, and it cannot rot when someone forgets the marker.

    KexI: capabilities are interface data

    isFoul already crosses the module boundary in the KexI chunk on KexiExport:62, KexiMethod:85, and KexiTraitRequiredMethod:154. Capability sets go where it goes.

    1. A new field, not a widened one. foul stays a bit; capability is a separate concept. Each of those three structs gains a requiredCapabilities list alongside isFoul. No version bump: the schema has been reset to v1 and, until there is a release, a format change edits the format rather than bumping it.
    2. Interface hashing must include it. Otherwise a stale .beam silently passes a capability check it should fail — the worst failure mode for a feature whose point is that nothing is silent.
    3. The chain truncates at the package boundary and expands only when asked. Shipping a full acquisition chain for every export would bloat the chunk, so KexI carries the set. A cross-package error reads Pkg.loadConfig requires FS.File and expands into the dependency's own chain only on request, re-derived from source (Tey always has it).

    Naming: capability, and what it explicitly does not mean

    The keyword is capability. It is consistent with docs/compilation.md:29, which already uses the word for exactly this concept — what a module needs and what a target provides.

    It is not the object-capability sense, and we are not going to claim that property. In ocap, authority travels only by unforgeable reference: code not handed a filesystem cannot reach one, and authority can be attenuated and revoked. That guarantee is unimplementable on BEAM — any process can call file:read_file/1, Kex.Intrinsic.File.read is reachable from anywhere, and the Erlang FFI (#83) will make Erlang.file.read_file reachable too. A capability system bypassable by one FFI call is a lint that looks like a security boundary, which is worse than not having one.

    So the property here is substitutability and presence — which implementation backs this module, and does this target have one — never authority. Ambient authority is preserved by design: the real module is the default, available everywhere, and with only substitutes it.

    Alternatives weighed and rejected:

    Name Why not
    port Conceptually ideal (ports and adapters), but BEAM ports are a core platform concept and Process.run literally uses one.
    backend Means codegen backend throughout the tree (68 files).
    driver Collides with Erlang port drivers; driver ENV reads badly.
    provider Collides with receiverProviders in KexI; vague and DI-flavoured.
    service On this platform implies a running process / GenServer.
    resource Implies an instance you acquire and release, not a namespace of operations.
    extern Reserved for the Erlang FFI (extern module).
    facility Genuinely unused and neutral — the fallback if capability is ever reconsidered.

    One rename lands with this, to remove the only real internal overload: ReadCapability / WriteCapability in src/stdlib/fs.kex:4-5 become ReadPermission / WritePermission. Those are permissions on a handle you already hold, not capabilities in either sense. The constructors (CanRead, CannotWrite) are unchanged, and they are what actually appears in signatures.

    It is also not dependency injection

    Worth stating because the two will be confused. In DI, a component declares its dependencies and an external assembler supplies them. Here nothing is declared, nothing is handed over, and there is no assembler — the code names FS.File and the name resolves to whatever the enclosing scope binds. Calling it DI imports a container model (scopes, lifecycles, singletons, wiring) that has no home in this design.

    Kex will have both, and they should keep separate names: #175 (trait-typed default parameters) is literally constructor injection and is small and independent; this is capabilities.

    Cost

    The implicit argument is a real argument, so this is not free. Being explicit about the bill:

    What is paid: one extra parameter threaded through call chains that reach a capability. Outside any with the value is the module default, a compile-time constant — but compilation is not whole-program (consumers compile against KexI), so a library cannot assume no future consumer will call it under a with. Either the parameter is always passed, or the default entry point becomes a thin wrapper over a parameterized form. Genuine elimination would need monomorphizing every function in every capability-reaching chain, which trades runtime for code size on a platform that loads modules whole.

    What bounds it: the purity rule, structurally. A function acquires a capability only by reaching a capability module, capability modules are effectful, and pure functions cannot call foul ones — so the acquiring set is a strict subset of the foul set. In the stdlib today that is 108 foul against 938 let, concentrated in six modules that are mostly thin leaf wrappers over intrinsics rather than deep chains (time.kex is 18 foul / 259 let; optionparser.kex is 2 / 56). Pure code — where the hot loops live — takes no parameter, ever. 21 of those 108 are mock.kex, which this design deletes.

    Applications are shaped differently from the stdlib: a request handler is foul from its entry point down, so chains are deeper there than these numbers suggest. The proportion still holds, since the parsing, validation and rendering underneath stay pure.

    Compared to the alternative: a process-dictionary read per capability call is worse than passing a term. But "cheaper than the design we are not choosing" is not "free", and the arity inflation on a real effectful chain is worth measuring before committing.

    The open problem is diagnostics, and it is the whole user interface

    What "acquisition chain" means concretely

    Today computeTransitiveEffects builds m_callGraph as name → set of called names and propagates a bool. The calls set holds names only, no source locations. It computes that a function is foul, never why.

    The chain means propagating the reason alongside the fact — one witness edge per (function, capability):

    error: `main` requires capability `FS.File`, not provided by target wasm
      main         → startServer   app.kex:2
      startServer  → loadConfig    server.kex:31
      loadConfig   → FS.File.read  config.kex:40
      FS.File declared a capability at src/stdlib/fs.kex:8
    

    Without it the same error is "main requires FS.File, which target wasm does not provide" — useless, since main is two lines and neither mentions a file.

    Three pieces of work: the fixpoint records a predecessor edge instead of OR-ing a bool; the error formatter walks the edges backwards; hover renders the same walk. One witness path suffices — if removing that call still fails you get a second error naming the next, exactly as rustc does with Send.

    Declaring the capability on the module also gives the chain a real endpoint (src/stdlib/fs.kex:8) instead of a compiler-internal one ("reaches intrinsic kex_fs_read").

    Why this is the risk and not a polish item

    In a design with no annotations anywhere — the entire point of it — the error message is the only place the requirement is ever visible. Scala at least makes you write (using fs: FS) at each level, so the source can be read. We are deliberately removing that. If the chain is deferred as "ship the checking, improve errors later", the feature ships with no user interface at all, and it is worse than Scala rather than better.

    How this scores against Scala's actual problems

    Scala's root cause is type-directed search over an open universe. Most of what follows is structurally impossible here, because resolution is name-directed lexical binding — FS.File is a name, and the lookup is "is there a with binding in an enclosing scope, else the module default":

    • Type-directed search — gone.
    • Ambiguity and import juggling — gone. One name, one binding, ordinary shadowing.
    • Incoherence (two instances, both compile, behave differently) — gone. No instance selection exists.
    • Implicit conversions — not in the design.
    • One keyword doing four jobs — avoided, but by discipline: capability, foul, and trait dispatch must stay separate concepts. Scala 3 spent a major version undoing that conflation.

    What survives is transitive acquisition, and on that single axis this is worse than Scala 3, which declares the acquisition at each level. Mitigations: the closed-but-extensible vocabulary (capabilities are declared modules, so every one in a program is enumerable — Scala's universe grows with every import), the acquisition chain above, and showing it at rest.

    Scala's problems were semantic and therefore unfixable by better messages. Ours reduces to one diagnostics problem. That is a much better position, conditional on actually building it.

    Editor support

    src/lsp/server.cxx already renders the foul marker into hover (:1135, :2173, :2358), so capability sets are the same rendering path carrying a set instead of a bit. "Displayed, not annotated" is an extension of working code, not vapor.

    The valuable surface is the chain, not the marker:

    foul loadGreeting(path: String) -> String
    requires FS.File
      ← loadGreeting calls readConfig    app.kex:12
      ← readConfig calls FS.File.read    config.kex:40
    

    Gap: the server advertises only completion, hover, definition, and references (server.cxx:1775). No inlay hints, so capabilities would show when you point at something, not at a glance while reading.

    Not foul(FS)

    An earlier version of this write-up proposed widening foul into a capability set. Rejected. foul stays a bit, capability and effect stay separate concepts, and no capability is ever named in a signature.

    Blockers and related

  3. added this to the 0.4.0 milestone on Aug 21, 2026
  4. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    Status: the runtime half is implemented and spec'd; the static half is not

    Working on the capabilities branch. spec/capabilities.kex covers all of it.

    capability Clock do
      foul now() -> Integer do return 1000 end
    end
    
    make FrozenClock, implement: Clock do
      foul now() -> Integer = @at
    end
    
    foul stamp() -> String = "at ${Clock.now()}"      # declares nothing, threads nothing
    foul deep()  -> String = "deep -> ${stamp()}"
    
    with Clock = FrozenClock { at: 7 } do
      IO.printLine(deep())        # deep -> at 7
    end
    

    deep and stamp are defined far from any with, and Clock.now() is written exactly as it would be in production. That was the design's whole requirement and it holds.

    What landed

    • capability and with keywords.
    • ModuleDef.isCapability — a capability parses as the ModuleDef every later phase already understands, so collect/resolve/typecheck/evaluator needed no per-construct branching. This is the "one keyword edit" claim realized in the compiler, not just in the stdlib.
    • WithExpr through parser, cloner, and Kex.AST reflection (a With variant in src/stdlib/kex/ast.kex).
    • registerCapabilityTraits — synthesizes the trait from the capability's own function signatures, so make Fake, implement: Clock works with existing machinery and the bodies stay the default.
    • Evaluator substitution, restored on any exit from the region including a raised error.
    • Per-process bindings in the scheduler.

    The process semantics are the ones argued for above, and they needed a real fix

    The first implementation used a single evaluator-wide stack. It leaked across processes — a with in one process changed what another process's code meant, which is precisely the invisible-lying property docs/testing.md refuses. Process::savedEnv was the precedent: bindings are now per-process, snapshotted at spawn, and saved/restored around each resume(). Verified in the spec:

    spawned inside:  at 7      # child inherits the replacement
    spawned outside: at 1000   # a process spawned outside the region never sees it
    

    No allow/3, and no decision needed about a child outliving the scope — it holds its own snapshot.

    Forced rename

    with collided with a real stdlib function: src/stdlib/type.kex:34 defined let with(name, args). Making it a keyword silently broke the Type module, surfacing as an unrelated-looking LSP hover failure. Renamed to Type.generic, which reads better beside the existing Type.named anyway. One spec call site updated.

    Not implemented — the whole static half

    • No requirement inference. Nothing computes which capabilities a function needs, so nothing is checked and nothing is displayed.
    • No acquisition chain, which this issue's thread identifies as the feature's only real user interface. Until it exists the design has no way to explain itself.
    • No target checking, so docs/compilation.md:29-46 is still unimplemented.
    • No KexI field — capability sets do not cross the module boundary.
    • No BEAM codegen for either construct. spec/capabilities.kex carries a new # kex: skip-beam marker, which is exactly the kind of Makefile-grepped convention this design is meant to retire.
    • implement: will not take a dotted name, so a capability nested in a module (FS.File) cannot be implemented yet. That blocks converting the actual stdlib, and therefore blocks deleting Mock.*.

    So what exists is the substitution mechanism and its process semantics. What does not exist is everything that makes a capability tracked rather than merely replaceable.

    Blocker cleared

    #172 is fixed on both backends, so trait-dispatched foul is now checked — the premise this design rests on.

  5. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    Requirement inference and the acquisition chain now exist

    The largest missing piece of the static half is in: the compiler computes which capabilities each function transitively needs, keeps the witness that explains each one, and tooling displays it.

    capability Clock do
      foul now() -> Integer do return 1000 end
    end
    
    foul stamp() -> Integer = Clock.now()
    foul deep()  -> Integer = stamp()
    

    Hovering deep in the editor:

    foul deep() -> Integer
    requires Clock
      ← deep calls stamp
      ← stamp uses Clock
    

    Nothing in deep's source mentions Clock — that is the whole design — so hover is the only place a reader can learn it, which is why this thread called the chain the feature's real user interface rather than a nicety.

    How it is built

    Requirements propagate over the same call graph and the same fixpoint shape the foul bit already uses (computeTransitiveEffects), which is what this issue anticipated. The difference is that each entry keeps the callee that carried the requirement in — "" when the function reaches the capability itself — so one witness per (function, capability) is enough to walk the chain backwards. Verified on a three-level chain, on two capabilities converging on one function, and on a nested capability (FS2.File); a function that needs nothing does not appear at all.

    The chain walk guards against a cyclic call graph, since mutual recursion must still terminate.

    What this unlocks

    Target checking is now mostly a matter of comparing this set against what a target provides — the inference and the explanation it needs both exist. docs/compilation.md:29-46 remains unimplemented, but no longer for want of the analysis.

    Still missing

    • Target checking itself, and any way for a target to declare what it provides.
    • The KexI field — capability sets do not cross the module boundary, so the requirement is only known within a compilation unit.
    • BEAM codegen for capability / with.
    • Converting the stdlib to capability blocks and deleting Mock.* (unblocked by implement: does not accept a qualified name: make Fake, implement: FS.File #179, not started).

    Covered by a new hover test in tests/lsp_test.cxx, checked against a deliberately wrong expectation to confirm it fails when it should.

  6. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    Stdlib conversion is blocked on the KexI field, and trying it proved why

    Attempted the flagship conversion — FS.File from module to capability, a one-word change, and the module is already correctly shaped (every member foul, every one an intrinsic, with the pure FS.Path deliberately outside the boundary).

    Two things came out of it.

    It found real threading gaps, which are fixed

    Marking a stdlib module as a capability turns on context threading for the whole prelude build, which is a far better harness than any hand-written test — it exercises every call shape at once. It immediately named two that were still bypassing the context:

    kex_prelude: undefined function 'DateTime.utcNow'/0 in 'Time.utcNow'/1
    kex_prelude: undefined function 'Process.run'/2 in 'Process.exec'/3
    

    Both are module-qualified calls to a sibling module member, emitted through callE("", ...) rather than the single localCall choke point. Every such site now routes through localCallExpr, so the context is appended in exactly one place. These fixes are kept — they are inert when no capability is declared, and they complete the threading.

    But the conversion itself cannot land yet

    A stdlib capability is invisible to a user program:

    error: 'FS.File' is not a capability, so it cannot be replaced by 'with'
    error: Unknown trait: FS.File
    

    registerCapabilityTraits walks the AST of the program being compiled. A capability declared in an imported module reaches the consumer through the KexI chunk, which carries no notion of capability-ness — so with FS.File = ... in a user program cannot see that FS.File is substitutable, and make Fake, implement: FS.File cannot see the generated trait.

    That is exactly the KexI field deferred earlier in this thread. It is a harder blocker than the nested-module lowering was, because it is not a lowering bug — the information genuinely does not cross the module boundary.

    So the ordering is: KexI first, then the stdlib conversion, then deleting Mock.*. The conversion is otherwise ready — FS.File needs one keyword, and #182 cleared the nested-call lowering that would have broken it.

    src/stdlib/fs.kex is reverted for now; turning the keyword on without the KexI half only costs prelude threading and buys nothing.

  7. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    Capability-ness now crosses the module boundary, and a stdlib capability provably works

    The KexI field is in, and it was the right diagnosis: a consumer reads its stdlib interfaces from the KexI chunk, not from source (kexi_registry.cxx:656), so without carrying the flag there a stdlib capability was invisible to every importer.

    • KexiStructuralMetadata.isCapability, serialized and deserialized (no version bump — the schema is reset to v1 and a format change edits the format until there is a release).
    • Populated in collect_metadata.cxx by finding the ModuleDef the chunk describes.
    • Consumed in kexi_registry.cxx into ImportedModuleInterface.isCapability, plus the same flag on the from-source path in prelude_interfaces.hxx.
    • registerCapabilityTraits now also registers imported capabilities and synthesizes their trait from the imported signatures, so make Fake, implement: FS.File and with FS.File = ... both resolve against a capability the program did not declare.

    Proof. Marking FS.File as a capability — a one-word change — and replacing it from a user program:

    from the fake     # FS.File.read, replaced
    true              # FS.File.exists?, replaced
    false             # outside the region: the real filesystem is back
    

    That is the goal of this issue, working, with no global mock state involved.

    But the conversion is reverted, for a reason worth recording

    Turning FS.File into a capability breaks four REPL/filesystem tests on BEAM:

    kex_kex_repl_session: undefined function read/2 in main/0
    

    The capability dispatch emits an override branch that calls the implementation through a local receiver dispatcher. When no type in that compilation unit implements the capability — the REPL session module, or the stdlib itself — the dispatcher does not exist and erlc rejects the module, even though the branch is unreachable.

    This is structural, not a tweak. A capability call site lives in one unit (the stdlib) while the implementation that replaces it lives in another (the test). The override branch therefore needs a dispatch that resolves across units — a runtime receiver dispatch on the value — rather than a compile-time local symbol.

    The same shape showed up earlier with a capability nobody implements, and I misread it then as a nested-module problem. It is the same root cause.

    So: the walker path is complete and the stdlib conversion works there. BEAM needs cross-unit override dispatch before FS.File can carry the keyword. src/stdlib/fs.kex is reverted; everything else here is kept, since the KexI plumbing is correct and independently needed.

  8. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    Dispatch consolidated; prelude capabilities need one more path

    The three hand-written capability-dispatch sites are now one helper, capabilityDispatch, with a lowered-args entry point for callers that build their argument slots before knowing the target is a capability. Each site supplies only how to build its own direct call for the miss case. Suite green: ctest 19/19, spec 277/0, spec-beam 262/0.

    This was the refactor to do before adding a fourth site, and it makes adding one a few lines rather than a copy.

    The remaining gap, stated precisely

    FS.File works end to end because FS is opt-in: using FS merges its source into the unit, so its calls resolve through moduleFunctions, where the dispatch has always been.

    A prelude capability does not. Marking ENV a capability, the walker substitutes correctly and BEAM does not — ENV.get("HOME", "(unset)") reaches neither the module path, nor the external receiver path, nor the qualified-call path I added the dispatch to. It takes a fourth route that has not been identified.

    The symptom is the one that matters: with ENV = FakeEnv { … } compiles, runs, and silently uses the real environment. No error, no warning. That is the fail-open this whole design exists to remove, so a prelude capability must not be marked until the path is found and covered.

    Suggested next step

    Do not hunt for it by trial — that is what cost most of this work. Dump Core for a two-line program (ENV.get("HOME", "x") under -e) and read which call is emitted; the emitted module and function name identify the branch directly. Every wrong hypothesis in this thread was settled that way in one step, and every trial loop took many.

    Once found, it is a call to the shared helper like the other three.

    Deleting Mock.* waits on this, since Mock.ENV, Mock.IO and Mock.Http all cover prelude modules. Mock.FS could be replaced today.

  9. akoskovacs commented on Aug 22, 2026

    @akoskovacs
    CollaboratorAuthor

    Prelude capabilities work — Mock.* is now replaceable

    ENV is a capability in src/stdlib/env.kex, substitutable from user code on both backends. Suite green: ctest 19/19, spec 278/0, spec-beam 263/0, prelude 462/0 and 453/0, stdlib 59/0.

    foul home() -> String = ENV.get("HOME", "(unset)")   # declares nothing, threads nothing
    
    with ENV = FakeEnv { vars: {"HOME": "/fake/home"} } do
      IO.printLine(home())        # /fake/home
    end
    IO.printLine(ENV.has?("..."))  # the real environment again
    

    Covered by spec/capability_prelude_env.kex, alongside spec/capability_stdlib_fs.kex for the opt-in case. Both run on both backends.

    The route that was missing

    Reading the emitted Core settled it in one step, after several wrong guesses. ENV.get(...) is a MethodCall whose outer guard passes, but it returns before reaching the receiver path — it is handled by the "External loaded modules" branch inside the module-candidates block (BinaryTree.fromList → 'Kex.BinaryTree':fromList). That branch had no dispatch.

    This is why the distinction between the two stdlib cases matters: FS is opt-in, so using FS merges its source and its calls resolve through moduleFunctions where the dispatch already lived. A prelude module only ever reaches a consumer through the compiled interface, on a different route entirely.

    All four dispatch points now call one shared capabilityDispatch helper — each supplies only how to build its own direct call for the miss case. That consolidation is what made adding this one a few lines.

    What is now unblocked

    Mock.FS, Mock.ENV, Mock.System, Mock.Http and Mock.IO can all be replaced by with, since both the opt-in and prelude routes work. That means:

    • rewriting the specs that use Mock.* (spec/fs.spec.kex, spec/mock_env.spec.kex, spec/prelude/io.spec.kex, spec/prelude/http.spec.kex, spec/system_os.spec.kex, spec/testing_dsl.spec.kex, spec/mock_io_spawn.spec.kex, spec/mock_gate.kex),
    • converting System, Http and IO to capabilities,
    • deleting src/stdlib/mock.kex, the mock intrinsics, and the test-gate machinery (--allow-mocks, the spec-file detection, and the matching runtime denial on both backends).

    That gate exists only because Mock.* lies globally and invisibly. A with block is lexically visible and cannot leak, so the whole apparatus goes away rather than being ported.

  10. akoskovacs commented on Aug 23, 2026

    @akoskovacs
    CollaboratorAuthor

    Status after #178

    Landed.

    • capability / with on both backends. Bindings are per-process and snapshotted at spawn, so a replacement reaches code that declares no dependency and threads no parameter, and never leaks into a process spawned outside the region.
    • Stdlib converted: FS.File, FS.Directory, ENV, Http and IO are capabilities. IO was last and needed two lowering fixes — a capability's interface is now its foul members, and capability dispatch carries trait dictionaries (IO.printLine(msg: Showable) is the first member with a trait-constrained parameter).
    • The stdlib ships the fakes, as the "do not regress the ergonomic floor" section requires: Mock.Files, Mock.Env, Mock.Response, each with a callback (onRead, onGet) for when a test wants a rule rather than a fixture. Stubbing a read is still one line and needs no user-written trait or record.
    • Mock.System deleted — faking the reported OS only exercises branching, not the platform behaviour behind it, and it had exactly one caller: the spec that tested it.

    Not landed — and it is the problem this issue opens with.

    The stateful mocks still write global state: Mock.FS (113 call sites), Mock.IO (75), Mock.Http (36), Mock.ENV (23). So every consequence listed at the top still holds — before/after clearing pairs remain, a forgotten clear still leaks into the next test, and specs still cannot run in parallel. The --allow-mocks gate machinery remains across 10 files.

    Why the capability route does not finish it. A replacement is a value, so it cannot change. Write-then-read round trips, before/after hooks observing an artifact across separate blocks, and Mock.IO's output capture all need a store — which means a process-backed double, not a record. That is a different design from anything in this issue's proposal.

    The cheapest useful increment is untried. This issue's own last section — scope Mock state to the enclosing it block automatically, changing no signatures and adding no concepts — would delete every before { Mock.FS.clear() } pair, close the forgotten-clear leak and make specs parallel-safe, without capabilities being involved at all. It remains the most direct route to the stated problem and is worth doing independently.

  11. added a commit that references this issue on Aug 23, 2026
    754726c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions

      0