Repository navigation
Conversation
…ov-only option party
…lag and pre-declare its categories
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds code coverage to mm as a build variant. Adding
covto the target list instruments thecompiled languages, gives the build its own build-root and install-prefix so it never mixes with
a regular build, and enables a
coveragetarget that gathers the data a test run produced andrenders it as a terminal summary, a browsable HTML tree, and an lcov file.
Both steps take the same
--target, because the report lives under the coverage build-root andreads the binaries from the coverage prefix.
Back ends
The instrumentation and the reader are chosen from the C++ compiler.
The clang family uses LLVM source-based coverage:
-fprofile-instr-generate -fcoverage-mappingatcompile and link. This records regions and branches rather than lines and resolves templates
correctly, which matters for heavily-templated C++. Each instrumented process writes its own
.profraw; mm exportsLLVM_PROFILE_FILEonce for the whole build, so every test driver depositsa raw profile into the collection directory with no per-recipe plumbing. The report is produced by
llvm-profdata mergefollowed byllvm-cov report/show/export.gcc uses
--coverage, which drops.gcno/.gcdasidecars next to the objects; the report isproduced by
gcovr, which discovers those pairs and emits HTML and lcov in one pass. Because alibrary object's data lands in the staging tree while a driver's lands next to its in-tree source,
gcovr searches both roots.
Both back ends converge on lcov as the interchange format, so the HTML renderers and editors
downstream speak one language regardless of compiler.
flangis switched to the LLVM form to matchthe clang front ends; a fallback note records the gcov alternative if a build rejects the flags.
Report scope
The reporters are pointed at the code under test automatically, discovered in the model pass: every
shared library installed under the prefix, plus every compiled test-driver binary registered by the
test suites. The drivers matter because a header-only template library instantiates most of its code
into them rather than into any
.so— the pyre case, where a report built from the shared librariesalone would miss nearly everything it tests.
coverage.objectsremains as an escape hatch foranything mm cannot find on its own.
When the same inline or template function is instantiated across many binaries, llvm-cov reports
"functions have mismatched data" and skips the conflicting instances. This is expected for a
template-heavy suite and does not corrupt the totals; it is a mild undercount of the skipped
functions.
Header attribution
A test driver includes a library's headers from the install prefix, so without help the compiler
records the installed copy's path in the coverage data. That makes gcov's reporter (rooted at the
project) drop the headers, and makes llvm-cov attribute them to the prefix rather than to the tree
you edit. A
covbuild corrects this at compile time: for each library, mm emits a-fprofile-prefix-map(gcc) /-fcoverage-prefix-map(clang) that rewrites the library's installedheader directory back to its source directory. The maps come straight from the library model — one
per library — and are ordered general-to-specific so the compilers' last-match-wins rule attribu 8000 tes
each header to the correct source tree.
The contribution is wired through
compiler.option.sourcesas acoverage.<language>party, addedonly when
covis a selected target and the language's compiler instruments (itscovflag isnon-trivial), so a non-coverage build — and any language that does not do coverage — is untouched.
The effect is that header coverage lands on the actual source files, both back ends agree, and the
lcov the editor reads points at the tree you have open.
Changes
make/compilers/clang/{clang,clang++,flang}.mm,make/compilers/gcc/{gcc,g++,gfortran}.mm— thecovflag for the clang toolchain becomes the LLVM source-based form (gcc keeps--coverage), andeach compiler names its per-compiler coverage prefix-map flag (
-fcoverage-prefix-map/-fprofile-prefix-map). The gcc-family clean now also sweeps the.gcno/.gcdasidecars acovbuild drops beside an object.
make/coverage/{init,model,rules}.mm— a new model class, registered inmake/merlin.mmahead ofprojectsso its option-source contributions exist before the workflows bake.model.mmresolvesthe back end, exports
LLVM_PROFILE_FILE, discovers the profiles / shared libraries / test drivers,builds the per-library header prefix-map, and defines the
coverage.<language>option party.rules.mmcarries the report, info, and clean recipes. Each reporter refuses to run outside acovbuild and fails with a one-line instruction when its tool is absent.
make/compilers/init.mm—compiler.option.sourcesadds thecoverage.<language>party under acov-and-instruments guard.etc/bash_completion/*,etc/zsh_completion/*— the three new global targets (coverage,coverage.info,coverage.clean) registered in both completions, with the test suites updated.docs/coverage.md— usage, back ends, report scope, header attribution, targets, tools, and editorintegration.
Targets
coveragecoverage.infocoverage.cleancoverage.cleanmatters between measurement runs: the profiles accumulate, so a fresh run shouldstart from a clean slate.
Tools
The reporters are not part of a compiler install and must be on the
PATH; they are invoked onlywhen the
coveragetarget runs.llvm-profdata,llvm-cov— conda-forgellvm-tools; Ubuntullvm.gcovr— conda-forgegcovr; Ubuntugcovr.gcovships withgcc.For editors, the lcov drives VS Code's Coverage Gutters (
ryanluker.vscode-coverage-gutters);coverage.infoprints its path. Because header coverage is attributed to source, the gutters lightup on the files in your working tree, not on installed copies under the prefix.
Verification
Exercised end to end on a full
pyre.libbuild and test run — a heavily header-only, template-drivenlibrary — across both back ends and both platforms:
leakage; header-only code (
lib/pyre/grid,lib/journal, …) present.leakage; driver auto-discovery and the staging-tree search recover the full surface.
(functions counted roughly triple on pyre.lib once the drivers are included).
(clang, gcc) × (cov, no-cov)matrix produces zero undefined-variable warnings.Notes on measurement
Coverage records execution, not correctness: a line that runs but computes the wrong value reports as
covered. For header-only templates it sees only the instantiations the tests create, so a percentage
measures those instantiations rather than the full API surface. It earns its keep on unreached error
paths, unexercised branches, and rank edge cases.