8000 make/extern: follow package .dependencies to compute a project's external dependencies by aivazis · Pull Request #40 · aivazis/mm · GitHub
[go: up one dir, main page]

Skip to content

make/extern: follow package .dependencies to compute a project's external dependencies - #40

Merged
aivazis merged 2 commits into
mainfrom
topo
Jun 7, 2026
Merged

aivazis merged 2 commits into
mainfrom
topo

Conversation

@aivazis
@aivazis aivazis commented Jun 7, 2026
Copy link
Copy Markdown
Owner

Summary

Makes package .dependencies load-bearing. An asset's external-dependency list is now the transitive closure of its declared externs — following each package's .dependencies — narrowed to what is actually installed. A package can therefore express an induced dependency (e.g. a parallel hdf5 requiring mpi) and have it reach every consumer's compile and link lines automatically, replacing the old pattern where projects hand-coded ${if ${findstring mpi,$(hdf5.parallel)},mpi} in their own config.

The problem

Projects used to reach into a package's internals at read time, e.g. in pyre.mm:

pyre.lib.extern := ... hdf5 ${if ${findstring mpi,$(hdf5.parallel)},mpi}

But hdf5.parallel is set by hdf5/init.mm, which is loaded by extern.load after all project configs are read and assets booted. So $(hdf5.parallel) was undefined when the conditional ran:

  • the clause was effectively dead (it only worked via a make command-line override), and
  • under -warn-undefined it emitted undefined variable 'hdf5.parallel'.

Root cause: the property driving the dependency decision is owned by the package and known only at load time, yet the decision was being made by the consumer before the package loaded. Meanwhile .dependencies was purely documentary — declared by every package but never walked.

What changed

Foundation (70fb839)

  • extern.closure {roots} — a 3-color DFS over .dependencies returning the transitive closure in link order (each dependent ahead of what it requires). A gray hit is a real cycle (warn); a black hit is a shared dependency (dedup). Prepend-on-finish yields link order directly, no separate reverse pass.
  • extern.deps.of — reads .dependencies via a ${flavor} probe so the walk tolerates tokens that never set the field (internal libraries, externs without deps) without tripping -warn-undefined.
  • extern.load is now a DFS (extern.load.one): it includes a package's config, then follows its .dependencies — so a load-time edge (parallel hdf5 electing mpi) is already defined by the time we recurse into it. Defaults .dependencies, .markers.required, and the hint to empty per loaded package.
  • hdf5.dependencies = ${if ${findstring mpi,$(hdf5.parallel)},mpi} — the induced edge now lives in the package that owns the knowledge, evaluated lazily so it sees the effective hdf5.parallel.

Resolution (d598e6e)

  • extern.resolve {asset} runs post-load for every asset and recomputes the tiers:
    • .extern.requested = closure of the declared .extern
    • .extern.supported = requested ∩ mm-configurable
    • .extern.available = supported ∩ installed (<pkg>.dir resolves), in link order
    • then folds internal deps + the installed external closure back into .extern (computed once) so the build recipes consume it unchanged — no recipe edits anywhere, and tests/rules.mm benefits for free.
  • Availability gate (<pkg>.dir): a package in the closure but with no install is dropped from .available. The induced-but-uninstalled case — e.g. a parallel hdf5 with no mpi installed — raises a self-describing complaint, distinguished from directly-declared shortfalls (left to extern.verify).
  • Severity knob — extern.complain routes complaints through extern.complain.severity (default warning) using literal warning/error function names. Warn-and-continue today so a real authority (the compiler/linker) faces the impossible config; flip to error to fail early. extern.verify audits independently and is never gated by this.
  • The per-package verify report gains a dependencies line and an unavailable-dependencies check.

make gotchas fixed along the way (recorded for future work)

  • Comments inside a define body are literal, not stripped — # all done leaked into the expansion. Fixed by moving comments out and invoking defines via ${eval ${call …}}.
  • A multi-line ${foreach} at the top level of a makefile is read line-by-line and breaks (unterminated foreach); it must live inside a define, invoked on one line.
  • The indirect-function trick ${$(severity) …} does not work — make resolves the function name at parse time, so it degraded to an undefined-variable reference. Replaced with ${if ${filter error,…},${error …},${warning …}}.
  • A bare ${call somedef,…} at top level leaks the define's trailing comment and body whitespace into the parse stream â 8000 †’ missing separator. Invoke via ${eval ${call …}}.

Consumer changes (separate repos, by the author)

pyre.mm, qed, and qef dropped their hand-coded ${if ${findstring mpi,$(hdf5.parallel)},mpi} clauses; the closure now supplies mpi.

Verification

Against real pyre via ~/dv/mm/mm:

hdf5.parallel pyre.lib.extern h5.ext.extern
off journal.lib hdf5 journal.lib python pybind11 hdf5
mpi journal.lib hdf5 mpi journal.lib python pybind11 hdf5 mpi

journal.lib (internal) is preserved; mpi is pulled in transitively only when hdf5 is parallel; extern.verify exits 0; no undefined-variable warnings originate from mm.

Notes / follow-ups

  • Topological order matters only for static archives; for shared libraries (the common case) link order is immaterial. The closure is a valid topo order regardless, though independent packages may be reordered.
  • hdf5.parallel still defaults to off and is user/config-set. Detecting it from the conda pkgdb (the mpi_* build variant) is a natural follow-up, so even a detected parallel build would induce mpi.

aivazis added 2 commits June 7, 2026 22:53
…make extern.load a DFS that follows .dependencies; hdf5 induces an mpi edge when parallel
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0