8000 Code coverage support by aivazis · Pull Request #51 · aivazis/mm · GitHub
[go: up one dir, main page]

Skip to content

Code coverage support - #51

Merged
aivazis merged 11 commits into
mainfrom
cov
Jul 22, 2026
Merged

aivazis merged 11 commits into
mainfrom
cov

Conversation

@aivazis
@aivazis aivazis commented Jul 21, 2026 •
Copy link
Copy Markdown
Owner

Summary

Adds code coverage to mm as a build variant. Adding cov to the target list instruments the
compiled languages, gives the build its own build-root and install-prefix so it never mixes with
a regular build, and enables a coverage target that gathers the data a test run produced and
renders it as a terminal summary, a browsable HTML tree, and an lcov file.

mm --target=cov,shared <suite>      # build and run the tests with instrumentation
mm --target=cov,shared coverage     # merge the data and render the report

Both steps take the same --target, because the report lives under the coverage build-root and
reads the binaries from the coverage prefix.

Back ends

The instrumentation and the reader are chosen from the C++ compiler.

The clang family uses LLVM source-based coverage: -fprofile-instr-generate -fcoverage-mapping at
compile and link. This records regions and branches rather than lines and resolves templates
correctly, which matters for heavily-templated C++. Each instrumented process writes its own
.profraw; mm exports LLVM_PROFILE_FILE once for the whole build, so every test driver deposits
a raw profile into the collection directory with no per-recipe plumbing. The report is produced by
llvm-profdata merge followed by llvm-cov report / show / export.

gcc uses --coverage, which drops .gcno/.gcda sidecars next to the objects; the report is
produced by gcovr, which discovers those pairs and emits HTML and lcov in one pass. Because a
library object's data lands in the staging tree while a driver's lands next to its in-tree source,
gcovr searches both roots.

Both back ends converge on lcov as the interchange format, so the HTML renderers and editors
downstream speak one language regardless of compiler. flang is switched to the LLVM form to match
the clang front ends; a fallback note records the gcov alternative if a build rejects the flags.

Report scope

The reporters are pointed at the code under test automatically, discovered in the model pass: every
shared library installed under the prefix, plus every compiled test-driver binary registered by the
test suites. The drivers matter because a header-only template library instantiates most of its code
into them rather than into any .so — the pyre case, where a report built from the shared libraries
alone would miss nearly everything it tests. coverage.objects remains as an escape hatch for
anything mm cannot find on its own.

When the same inline or template function is instantiated across many binaries, llvm-cov reports
"functions have mismatched data" and skips the conflicting instances. This is expected for a
template-heavy suite and does not corrupt the totals; it is a mild undercount of the skipped
functions.

Header attribution

A test driver includes a library's headers from the install prefix, so without help the compiler
records the installed copy's path in the coverage data. That makes gcov's reporter (rooted at the
project) drop the headers, and makes llvm-cov attribute them to the prefix rather than to the tree
you edit. A cov build corrects this at compile time: for each library, mm emits a
-fprofile-prefix-map (gcc) / -fcoverage-prefix-map (clang) that rewrites the library's installed
header directory back to its source directory. The maps come straight from the library model — one
per library — and are ordered general-to-specific so the compilers' last-match-wins rule attribu 8000 tes
each header to the correct source tree.

The contribution is wired through compiler.option.sources as a coverage.<language> party, added
only when cov is a selected target and the language's compiler instruments (its cov flag is
non-trivial), so a non-coverage build — and any language that does not do coverage — is untouched.
The effect is that header coverage lands on the actual source files, both back ends agree, and the
lcov the editor reads points at the tree you have open.

Changes

  • make/compilers/clang/{clang,clang++,flang}.mm, make/compilers/gcc/{gcc,g++,gfortran}.mm — the
    cov flag for the clang toolchain becomes the LLVM source-based form (gcc keeps --coverage), and
    each compiler names its per-compiler coverage prefix-map flag (-fcoverage-prefix-map /
    -fprofile-prefix-map). The gcc-family clean now also sweeps the .gcno/.gcda sidecars a cov
    build drops beside an object.
  • make/coverage/{init,model,rules}.mm — a new model class, registered in make/merlin.mm ahead of
    projects so its option-source contributions exist before the workflows bake. model.mm resolves
    the back end, exports LLVM_PROFILE_FILE, discovers the profiles / shared libraries / test drivers,
    builds the per-library header prefix-map, and defines the coverage.<language> option party.
    rules.mm carries the report, info, and clean recipes. Each reporter refuses to run outside a cov
    build and fails with a one-line instruction when its tool is absent.
  • make/compilers/init.mm — compiler.option.sources adds the coverage.<language> party under a
    cov-and-instruments guard.
  • etc/bash_completion/*, etc/zsh_completion/* — the three new global targets (coverage,
    coverage.info, coverage.clean) registered in both completions, with the test suites updated.
  • docs/coverage.md — usage, back ends, report scope, header attribution, targets, tools, and editor
    integration.

Targets

target effect
coverage merge the collected data and render summary + HTML + lcov
coverage.info show the resolved settings and the report paths
coverage.clean discard the collected data and the rendered report

coverage.clean matters between measurement runs: the profiles accumulate, so a fresh run should
start from a clean slate.

Tools

The reporters are not part of a compiler install and must be on the PATH; they are invoked only
when the coverage target runs.

  • llvm back end: llvm-profdata, llvm-cov — conda-forge llvm-tools; Ubuntu llvm.
  • gcov back end: gcovr — conda-forge gcovr; Ubuntu gcovr. gcov ships with gcc.

For editors, the lcov drives VS Code's Coverage Gutters (ryanluker.vscode-coverage-gutters);
coverage.info prints its path. Because header coverage is attributed to source, the gutters light
up on the files in your working tree, not on installed copies under the prefix.

Verification

Exercised end to end on a full pyre.lib build and test run — a heavily header-only, template-driven
library — across both back ends and both platforms:

  • clang / llvm (macOS arm64 and Linux x64): report attributes every record to source, zero prefix
    leakage; header-only code (lib/pyre/grid, lib/journal, …) present.
  • gcc / gcov (Linux x64): same — library and header coverage attributed to source, zero prefix
    leakage; driver auto-discovery and the staging-tree search recover the full surface.
  • Driver auto-discovery takes the report from library-only visibility to the full template surface
    (functions counted roughly triple on pyre.lib once the drivers are included).
  • A normal build is unaffected: coverage is inactive, no option party is added, and the
    (clang, gcc) × (cov, no-cov) matrix produces zero undefined-variable warnings.
  • The bash and zsh completion suites pass.

Notes on measurement

Coverage records execution, not correctness: a line that runs but computes the wrong value reports as
covered. For header-only templates it sees only the instantiations the tests create, so a percentage
measures those instantiations rather than the full API surface. It earns its keep on unreached error
paths, unexercised branches, and rank edge cases.

@aivazis
aivazis merged commit 4711eca into main Jul 22, 2026
@aivazis
aivazis deleted the cov branch July 22, 2026 07:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

0