Repository navigation
Conversation
…make extern.load a DFS that follows .dependencies; hdf5 induces an mpi edge when parallel
…rnal dependencies of a project
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes package
.dependenciesload-bearing. An asset's external-dependency list is now the transitive closure of its declared externs — following each package's.dependencies— narrowed to what is actually installed. A package can therefore express an induced dependency (e.g. a parallelhdf5requiringmpi) and have it reach every consumer's compile and link lines automatically, replacing the old pattern where projects hand-coded${if ${findstring mpi,$(hdf5.parallel)},mpi}in their own config.The problem
Projects used to reach into a package's internals at read time, e.g. in
pyre.mm:But
hdf5.parallelis set byhdf5/init.mm, which is loaded byextern.loadafter all project configs are read and assets booted. So$(hdf5.parallel)was undefined when the conditional ran:-warn-undefinedit emittedundefined variable 'hdf5.parallel'.Root cause: the property driving the dependency decision is owned by the package and known only at load time, yet the decision was being made by the consumer before the package loaded. Meanwhile
.dependencieswas purely documentary — declared by every package but never walked.What changed
Foundation (
70fb839)extern.closure {roots}— a 3-color DFS over.dependenciesreturning the transitive closure in link order (each dependent ahead of what it requires). Agrayhit is a real cycle (warn); ablackhit is a shared dependency (dedup). Prepend-on-finish yields link order directly, no separate reverse pass.extern.deps.of— reads.dependenciesvia a${flavor}probe so the walk tolerates tokens that never set the field (internal libraries, externs without deps) without tripping-warn-undefined.extern.loadis now a DFS (extern.load.one): it includes a package's config, then follows its.dependencies— so a load-time edge (parallelhdf5electingmpi) is already defined by the time we recurse into it. Defaults.dependencies,.markers.required, and the hint to empty per loaded package.hdf5.dependencies = ${if ${findstring mpi,$(hdf5.parallel)},mpi}— the induced edge now lives in the package that owns the knowledge, evaluated lazily so it sees the effectivehdf5.parallel.Resolution (
d598e6e)extern.resolve {asset}runs post-load for every asset and recomputes the tiers:.extern.requested= closure of the declared.extern.extern.supported= requested ∩ mm-configurable.extern.available= supported ∩ installed (<pkg>.dirresolves), in link order.extern(computed once) so the build recipes consume it unchanged — no recipe edits anywhere, andtests/rules.mmbenefits for free.<pkg>.dir): a package in the closure but with no install is dropped from.available. The induced-but-uninstalled case — e.g. a parallelhdf5with nompiinstalled — raises a self-describing complaint, distinguished from directly-declared shortfalls (left toextern.verify).extern.complainroutes complaints throughextern.complain.severity(defaultwarning) using literalwarning/errorfunction names. Warn-and-continue today so a real authority (the compiler/linker) faces the impossible config; flip toerrorto fail early.extern.verifyaudits independently and is never gated by this.dependenciesline and an unavailable-dependencies check.make gotchas fixed along the way (recorded for future work)
definebody are literal, not stripped —# all doneleaked into the expansion. Fixed by moving comments out and invoking defines via${eval ${call …}}.${foreach}at the top level of a makefile is read line-by-line and breaks (unterminated foreach); it must live inside adefine, invoked on one line.${$(severity) …}does not work — make resolves the function name at parse time, so it degraded to an undefined-variable reference. Replaced with${if ${filter error,…},${error …},${warning …}}.${call somedef,…}at top level leaks the define's trailing comment and body whitespace into the parse stream â 8000 †’missing separator. Invoke via${eval ${call …}}.Consumer changes (separate repos, by the author)
pyre.mm,qed, andqefdropped their hand-coded${if ${findstring mpi,$(hdf5.parallel)},mpi}clauses; the closure now suppliesmpi.Verification
Against real
pyrevia~/dv/mm/mm:hdf5.parallelpyre.lib.externh5.ext.externoffjournal.lib hdf5journal.lib python pybind11 hdf5mpijournal.lib hdf5 mpijournal.lib python pybind11 hdf5 mpijournal.lib(internal) is preserved;mpiis pulled in transitively only when hdf5 is parallel;extern.verifyexits 0; no undefined-variable warnings originate from mm.Notes / follow-ups
hdf5.parallelstill defaults tooffand is user/config-set. Detecting it from the conda pkgdb (thempi_*build variant) is a natural follow-up, so even a detected parallel build would inducempi.