Repository navigation
feature: CI refactor — clear build/test phases, three gating targets #687
Description
Summary
Comprehensive CI refactor for gominimal/minimal to eliminate redundant rebuilds, separate build from test phases across lanes, and gate merges on all three production platforms (macOS/HVF, Linux native minimald, Linux/KVM). The work is sequenced as 10 independently landable PRs (PR0–PR9) plus an admin ruleset flip.
Problem
The current CI has structural problems: test jobs rebuild artifacts instead of consuming pre-built ones; monolithic jobs mix setup, build, and e2e; shared steps are duplicated across workflows and release.yml copies rather than reuses CI logic; only lint and unit tests gate merges — no VM boot blocks a bad merge; the ci-docs-skip.yml inverse-path hack creates wedge and silent-ungate failure modes; and the Mac mini's time is spent on unit/build work that could run on GH-hosted runners.
Desired outcome
- One build per platform per run; test jobs consume pre-built artifacts
- Every lane reads as clear build → test phases; no monolithic jobs
- Shared steps (Rust setup, libkrun, guest artifacts, linkage rewrite) in one composite/script;
release.ymlreuses CI's setup - Three required gating targets:
ci-macos-success,ci-linux-native-success,ci-linux-kvm-success - Robust required-check mechanics via changes-filter jobs +
if: always()aggregators;ci-docs-skip.ymldeleted - Coverage gaps closed: native minimald daemon e2e,
crates/minimald/**triggering the mac lane, release artifacts smoke-tested before staging - Mac mini handles only hypervisor e2e; unit/build moves to GH-hosted runners
- Dev scaffolding removed: boop timeouts, split arm64 builds, dead conditionals, redundant verify workflows
Scope notes
Work is sequenced: PR0 (done) → PR1 (in review) → PR2–PR9 + Admin ruleset flip; each PR soaks before the ruleset flip makes lanes required. Key files: .github/workflows/ci.yml, ci-macos.yml, ci-linux-kvm.yml, ci-shell-installer.yml, ci-docs-skip.yml (delete), ci-netns.yml (delete), ci-gvproxy.yml (delete), release.yml, and new ci-linux-native.yml. Networking test surface stays mothballed by decision; arm64-Linux binaries remain untested by decision.
This issue is the tracking issue for this work. A maintainer with write
access starts the SDD pipeline by commenting /spec on this issue or by
flipping the sdd:spec label.
Steer the pipeline with these comment commands. Each is gated to commenters
with write access to the repository.
| Command | Where | Effect |
|---|---|---|
/spec |
this tracking issue | re-run sdd-spec to draft or revise the spec |
/fastpath |
this tracking issue | confirm a single-session change; sdd-spec produces a stub spec PR and an execution plan comment in one run (ADR 0012) |
/triage |
this tracking issue, after the spec PR is merged | start sdd-triage phase A (architecture) |
/approve |
this tracking issue | full path: confirm the proposed plan so sdd-triage creates the Unit and task sub-issues. Fast path: dispatch the execution plan against sdd-execute-{tier} |
/dispatch |
this tracking issue (full path only) | arm the cascade across the task tree |
/revise <note> |
a spec, architecture, or implementation PR, or this tracking issue | re-run the owning agent with the note as an added instruction |
/execute |
a task sub-issue | run sdd-execute for that task ahead of the schedule |
Merging the spec PR advances to the architecture phase; merging the
architecture PR advances to a plan-comment phase, where sdd-triage posts
the proposed plan as one comment on this tracking issue. When an agent needs
a human decision it applies the needs-human label and posts one comment;
clear the label once you have answered and the agent resumes.
Routed from #687 by inbox-route (https://github.com/gominimal/inbox/actions/runs/29043264519).
Routing-Key: inbox-route/I_kwDOSUhdos8AAAABIQgrMA
Original request
Tracking issue for the CI refactor of gominimal/minimal. Work is sequenced as independently landable PRs; each soaks before the ruleset flip makes lanes required.
Goals
- One build per platform per run — each binary/artifact compiled once, test jobs consume it; no rebuilds inside test jobs.
- Clear phase separation — every lane reads as build → test; no monolithic jobs mixing setup, build, and e2e.
- Zero duplicated definitions — shared steps live in one composite/script (rust setup, libkrun, guest artifacts, linkage rewrite);
release.ymlreuses CI's, never copies. - Three gating targets — macOS/HVF, Linux native minimald, Linux minvmd/KVM all required for merge (today only lint/unit gates; no VM boot blocks a bad merge).
- Robust required-check mechanics — always-triggered lanes with path-filter jobs +
if: always()aggregators; kills theci-docs-skip.ymlinverse-path hack and the wedge/silent-ungate failure modes. - Close coverage holes — native minimald daemon e2e (new),
crates/minimald/**triggering the mac lane, release artifacts smoke-tested before staging. - Test what ships — CI exercises production variants (own trimmed libkrun, @rpath linkage), not dev-only stand-ins.
- Protect scarce runners — the Mac mini does hypervisor e2e only; unit/build work moves to GH-hosted; flake reruns cost ~15 min, not an hour.
- Remove dev scaffolding — boop timeouts, split arm64 builds, dead conditionals, redundant verify workflows, mothballed networking proofs.
Ordered subtasks
- PR0 — mac e2e consolidation + redundant-step removal — done: gominimal/minimal#678
- PR1 — shared setup composites + linkage-rewrite script extraction (behavior-preserving) — in review: gominimal/minimal#685
- PR2 — consolidate macOS libkrun on our own trimmed build —
vendor/libkrun.lockpin,scripts/build-libkrun-macos.sh, cached viaactions/cache(GH-hosted) + versioned local dir (mini); CI boots the exact dylib that ships; brew tap dropped from CI; release builds via the same script. Merged: gominimal/minimal#694. Bonus catch: the e2e's combined cargo invocation had been feature-unifying libkrun into the CLI all along (masked by brew's absolute install name) — now split + guarded by a system-libraries assert. - PR3 — mac lane finish — GH-hosted
unitjob (clippy/test minvmd, sessions, minimal) replaces the mini'sbuild-macos;krun_smokemoves intoe2e; per-PR production-linkage check;add(path fix landed on main via feat(minvmd,minimald): per-VM writable /dev/vdb ext4 volume #672). Merged: gominimal/minimal#696. Note: the plannedcrates/minimald/**to pathscargo test -p minimalon macOS is impossible today — minimal's dev-deps pull minimald, whose ungated hakoniwa→libcgroups→procfs chain is Linux-only; needs a cfg-gating PR if wanted. - PR4 — new
ci-linux-native.ymllane — workspace tests (networking feature steps mothballed) + new native minimald daemon e2e (now the unifiedscripts/session-e2e.sh, shared by all target lanes); deleteci-netns.yml; born with changes-filter + aggregator. Merged: gominimal/minimal#697. The e2e drivesminimal activate+attach --command(non-interactive exec);E2E_ACTIVATE_ARGSis the drop-in for--loadout devonce feat(sessions): loadouts, config, and debug commands #686 lands. - PR5 — KVM lane gate-ready — changes-filter + aggregator; retries dropped entirely by owner decision — the single recorded apt transient (run 28897380911) doesn't justify the machinery; reintroduce only with fresh evidence; soak 1–2 weeks tracking flake rate. In review: gominimal/minimal#698.
- PR6 — KVM build/test split —
MINVMD_BINenv override in e2e harnesses (falls back toCARGO_BIN_EXE_minvmd),build-linux→test-kvmviatestbins.jsonmanifest; flake rerun drops from ~60 to ~15 min. Merged with PR5 via gominimal/minimal#698 (gominimal/minimal#699). Proven on the merge run: test-kvm reran in ~1 min on prebuilt harnesses. - N1 — pin the Rust toolchain —
rust-toolchain.toml@ stable 1.97.0; surfaced + fixed 6 clippy-1.97 lints across 6 crates. Merged: gominimal/minimal#701 - N2+N3 + KVM session-e2e — folded into one PR — nextest profiles (
ci/vm, no retries — own-ip attach intermittently hangs forever on AF_VSOCK missed wakeup #588 leak-wedge), KVM lane on nextest archives (testbins.json + run-testbin.sh deleted), unified session e2e wired into the KVM lane (DM3), daemon lifecycle extracted toscripts/lifecycle-e2e.sh, krun_add_disk3 assert moved into thesetup-libkrun-linuxcomposite (covers release.yml). Merged: gominimal/minimal#702. Proven live:session e2e OKon KVM (1653ms cold activate incl. microVM boot),daemon lifecycle OK. - N7 — auto-discovery is the default —
*_e2e/*_root_e2enaming convention (renamevsock_relay_e2e→vsock_relay_root_e2e,krun_smoke→krun_smoke_e2e), delete retiredbridge_e2e(owner-approved), lanes select by convention filtersets instead of hardcoded test lists, meta-test guards the convention. Adding a test never requires a CI edit. Merged: gominimal/minimal#732 — shipped as the honest integration-vs-e2e taxonomy: harnesses renamed*_integration/*_root_integration(not_e2e), lanes select bybinary(/_integration$/), crate meta-tests guard the suffix + document when to use an integration harness vs. a full-CLI e2e script; bonus: removed minvmd's dead host-side tap relay (net/relay.rs, no production caller since the relay moved into the guest) and scrubbed workflow YAMLs of internal plan jargon. - N4 — cache discipline — rust-cache
shared-keyper job class +save-ifmain-only; big raw caches → restore/save split;CARGO_PROFILE_DEV_DEBUG=line-tables-only;--lockedsweep. Before PR7. Merged: gominimal/minimal#703 - PR7 — trigger flip —
ci.yml/ci-macos.yml/ci-shell-installer.ymllose trigger-level path filters, gain changes-jobs + aggregators (incl. the newci-macos-success); deleteci-docs-skip.yml(atomic) andci-gvproxy.yml; gvproxy pin coverage moves into the KVM filter. Merged: gominimal/minimal#712 — the soak has started. - N5 —
nightly-tests.yml(merged: gominimal/minimal#720; +installer suite via workflow_call) — advisories watch, session-e2e soak ×10 (hunts the own-ip attach intermittently hangs forever on AF_VSOCK missed wakeup #588 wedge class), beta + cargo-update canaries (continue-on-error), actionlint/zizmor/cargo-machete hygiene, issue-on-failure notify; scheduled runs keep main's caches warm. Drive-by:timeout-minutesfor promote.yml's promote job. - N8 — CI strategy + agent/developer contract into repo docs (parked on branch
ci/docs-contract, unblocks after N7) —docs/ci-strategy.md(incl. §10),CONTRIBUTING.mdcontract table (test kind → location → local command; never edit.github/workflows/),CLAUDE.mdpointer,.github/CODEOWNERSfreeze on/.github/, junit +$GITHUB_STEP_SUMMARYproofs in core-tests. Merged: gominimal/minimal#742 — rebuilt fresh off main (the parkedci/docs-contracthad drifted): CONTRIBUTING.md contract + docs/ci-strategy.md updated to the shipped_integrationtaxonomy, CLAUDE.md pointer, nextest JUnit +$GITHUB_STEP_SUMMARY. Dropped the redundant nightly/soak/promote bits (already on main via ci: add the nightly test tier #720) and the CODEOWNERS (superseded by the repo-wide* @gominimal/minimalistsfrom chore: add CODEOWNERS for the minimalists team #728). - Admin — ruleset flip — required checks become
ci-success,ci-linux-native-success,ci-linux-kvm-success,ci-macos-success,ci-shell-installer-success(after soak; open PRs need a synchronize to pick up new contexts). Same session: enable "require code owner review" so N8's CODEOWNERS freeze on/.github/bites. Done 2026-07-14 (ruleset 14240442): required checks flipped fromci-successalone to all five lane aggregators (ci-success,ci-linux-native-success,ci-linux-kvm-success,ci-macos-success,ci-shell-installer-success);require_code_owner_reviewenabled so the repo-wide CODEOWNERS (* @gominimal/minimalists, chore: add CODEOWNERS for the minimalists team #728) gates.github/. Flipped on 1 green soak night (nightly startup bug fixed in fix(ci): grant the nightly installer job pull-requests: read #750) — watch the nightly soak; revert = restorerequired_status_checksto[ci-success]. - PR8 — remove
ci.ymltestjob — redundant once the native lane became required. Merged: gominimal/minimal#755. Removed the workspacetestjob (core-tests + its bespoke mTLS--features networking-proxystep) and droppedtestfromci-success.needs; the native lane's requiredtestsjob runs the identical suite. The proxy step was dropped, not moved — those tests are feature-gated inminimaldand should run wherever the feature is enabled, not via a hand-wired YAML step. - PR9 — release.yml overhaul (core) — Merged: gominimal/minimal#757.
verify-ciguard replaces the duplicated test job: it queries the merge commit's five required aggregators and refuses to release unless all are green (skip_ci_verifyinput bypasses with a warning). Addeddry_run(build/sign/assemble but publish nothing — no GCS upload, GitHub release, installer-bucket write, or channel advance). arm64 build recollapsed from three sequentialcross builds to onecross build --package minimal --package mip --package minimald+ single rename, timeout 90→60.--lockedon all five release builds. The mac release build stays on the self-hosted runner — the Developer ID signing keychain lives there (gominimal/minimal#680). Validated post-merge via adry_run+skip_ci_verifydispatch on main (run 29375276363). Deferred from this PR: the canonical-basename seam (aarch64/arm64) was left as-is — not needed for the gate work. - PR9b — smoke the shipped artifacts — download the release binaries and prove them: linux-amd64 daemon e2e + KVM boot + skip-tolerant mac installer-layout boot on the mini. Split out of PR9 core — only validatable via a real release dispatch, so it lands after the release path is exercised.
- N6 — darwin testbed: the mini stops compiling (last) — hosted
unitjob also archives-p minvmd -p sessions+ buildsminvmd/minimal, bundles a@loader_path-layout testbed withlibkrun.dylib; the mini downloads, codesigns, runs. Podman pattern; collapses the unit/e2e build divergence.
Notes
- Networking test surface (netns/mesh/proxy/wg proofs) is mothballed by decision — code stays in-tree, not CI-covered; the KVM gvproxy relay e2e stays.
- arm64-Linux binaries remain untested by decision — gap documented in-workflow.
gs://minimal-shimarchive upload + legacy dogfood installer stay (migration owned separately).
🤖 Generated with Claude Code
Activity
gominimal-aw-bot commentedon Jul 9, 2026
Routing triage — #687
Kind: epic | Route: sdd | Confidence: 0.95 | Target: gominimal/minimal
This is an epic tracking a phased CI refactor of gominimal/minimal, sequenced as 10 independently landable PRs (PR0–PR9) plus an admin ruleset flip. Two PRs are already done or in review. The work belongs in gominimal/minimal: the issue explicitly links PRs there, and every CI workflow file named in the plan was confirmed present in
.github/workflows/of that repo.Route: sdd — the scope spans 10+ workflow files across multiple PRs and requires design judgment (phase ordering, runner allocation, required-check mechanics); too large for patch.
Launch: a maintainer comments
/specon the work issue to start the SDD pipeline.Evidence
.github/workflows/): Every file named in the issue plan confirmed present:ci.yml,ci-macos.yml,ci-linux-kvm.yml,ci-shell-installer.yml,ci-docs-skip.yml,ci-gvproxy.yml,ci-netns.yml,release.yml.route: allow,sdd: true; the SDD pipeline is available.[inbox-route:decision]
{"schema":"inbox-route/v1","issue":269,"issue_node_id":"I_kwDOSUhdos8AAAABIQgrMA","dry_run":false,"kind":"epic","route":"sdd","confidence":0.95,"targets":[{"repo":"gominimal/minimal","reason":"all workflow files named in the plan exist in .github/workflows/ of this repo; the issue explicitly links PRs there and CI work is owned here"}],"labels":["kind:epic","priority:should-have"],"patch":null,"work_issue":{"title":"feature: CI refactor — clear build/test phases, three gating targets","body":"## Summary\n\nComprehensive CI refactor for gominimal/minimal to eliminate redundant rebuilds, separate build from test phases across lanes, and gate merges on all three production platforms (macOS/HVF, Linux native minimald, Linux/KVM). The work is sequenced as 10 independently landable PRs (PR0–PR9) plus an admin ruleset flip.\n\n## Problem\n\nThe current CI has structural problems: test jobs rebuild artifacts instead of consuming pre-built ones; monolithic jobs mix setup, build, and e2e; shared steps are duplicated across workflows and `release.yml` copies rather than reuses CI logic; only lint and unit tests gate merges — no VM boot blocks a bad merge; the `ci-docs-skip.yml` inverse-path hack creates wedge and silent-ungate failure modes; and the Mac mini's time is spent on unit/build work that could run on GH-hosted runners.\n\n## Desired outcome\n\n- One build per platform per run; test jobs consume pre-built artifacts\n- Every lane reads as clear build → test phases; no monolithic jobs\n- Shared steps (Rust setup, libkrun, guest artifacts, linkage rewrite) in one composite/script; `release.yml` reuses CI's setup\n- Three required gating targets: `ci-macos-success`, `ci-linux-native-success`, `ci-linux-kvm-success`\n- Robust required-check mechanics via changes-filter jobs + `if: always()` aggregators; `ci-docs-skip.yml` deleted\n- Coverage gaps closed: native minimald daemon e2e, `crates/minimald/**` triggering the mac lane, release artifacts smoke-tested before staging\n- Mac mini handles only hypervisor e2e; unit/build moves to GH-hosted runners\n- Dev scaffolding removed: boop timeouts, split arm64 builds, dead conditionals, redundant verify workflows\n\n## Scope notes\n\nWork is sequenced: PR0 (done) → PR1 (in review) → PR2–PR9 + Admin ruleset flip; each PR soaks before the ruleset flip makes lanes required. Key files: `.github/workflows/ci.yml`, `ci-macos.yml`, `ci-linux-kvm.yml`, `ci-shell-installer.yml`, `ci-docs-skip.yml` (delete), `ci-netns.yml` (delete), `ci-gvproxy.yml` (delete), `release.yml`, and new `ci-linux-native.yml`. Networking test surface stays mothballed by decision; arm64-Linux binaries remain untested by decision.\n\n---\n<!-- SDD command vocabulary. Do not edit this footer. -->\n\nThis issue is the tracking issue for this work. A maintainer with write\naccess starts the SDD pipeline by commenting `/spec` on this issue or by\nflipping the `sdd:spec` label.\n\nSteer the pipeline with these comment commands. Each is gated to commenters\nwith write access to the repository.\n\n| Command | Where | Effect |\n|---|---|---|\n| `/spec` | this tracking issue | re-run `sdd-spec` to draft or revise the spec |\n| `/fastpath` | this tracking issue | confirm a single-session change; `sdd-spec` produces a stub spec PR and an execution plan comment in one run (ADR 0012) |\n| `/triage` | this tracking issue, after the spec PR is merged | start `sdd-triage` phase A (architecture) |\n| `/approve` | this tracking issue | full path: confirm the proposed plan so `sdd-triage` creates the Unit and task sub-issues. Fast path: dispatch the execution plan against `sdd-execute-{tier}` |\n| `/dispatch` | this tracking issue (full path only) | arm the cascade across the task tree |\n| `/revise <note>` | a spec, architecture, or implementation PR, or this tracking issue | re-run the owning agent with the note as an added instruction |\n| `/execute` | a task sub-issue | run `sdd-execute` for that task ahead of the schedule |\n\nMerging the spec PR advances to the architecture phase; merging the\narchitecture PR advances to a plan-comment phase, where `sdd-triage` posts\nthe proposed plan as one comment on this tracking issue. When an agent needs\na human decision it applies the `needs-human` label and posts one comment;\nclear the label once you have answered and the agent resumes."},"related":[{"repo":"gominimal/minimal","number":238,"relation":"related"},{"repo":"gominimal/minimal","number":620,"relation":"related"}],"rationale":"The issue is an epic tracking a phased CI refactor of gominimal/minimal. The target is unambiguous: the issue body links directly to gominimal/minimal PRs (#678, #685), and the repo's .github/workflows/ directory contains every file named in the plan (ci.yml, ci-macos.yml, ci-linux-kvm.yml, ci-docs-skip.yml, ci-netns.yml, ci-gvproxy.yml, release.yml). Distillery confirms prior CI evolution in this project: release workflow split (gominimal/minimal#620, merged), macOS self-hosted runner setup (gominimal/minimal#283, merged), libkrun brew provisioning (gominimal/minimal#508, merged). The work spans 10+ workflow files across multiple PRs and requires design judgment (phase ordering, runner allocation, required-check mechanics), making sdd the correct route over patch. No duplicates or prior routing markers found in either inbox or minimal. No routing-outcome precedents in Distillery inbox project."}gominimal-aw-bot commentedon Jul 9, 2026
[inbox-route:ledger]
Routed from gominimal/inbox#269 by inbox-route (https://github.com/gominimal/inbox/actions/runs/29043264519)
Routing-Key: inbox-route/I_kwDOSUhdos8AAAABIQgrMA
Route: sdd
Work-Issue: #687 (in-place — the transferred issue is the work item)
sdd:spec: withheld — a maintainer launches via /spec or a sdd:spec label flip (a bot actor is refused by the target sdd-spec gate; #226)
[-]CI refactor (minimal): clear build/test phases, three gating targets[/-][+]feature: CI refactor — clear build/test phases, three gating targets[/+]ci(macos): consolidate libkrun on our own pinned source build (#694)
ci: one session e2e for every target; retire the mac autospawn test
ci: drop the apt retry wrapper
ci(linux-kvm): make the lane gate-ready — changes filter, aggregator,…
ci(linux-kvm): reshape the lane to its end state — nextest archive, u…
ci(linux-kvm): reshape the lane to its end state
ci: adopt nextest profiles; reshape the KVM lane to its end state (#702)
ci: cache discipline, slim CI debug info, and a --locked sweep (#703)
ci: flip every lane to always-trigger with in-workflow gates (#712)
ci: nightly test tier (advisories, session-e2e soak, canaries, hygiene)
ci: nightly test tier (advisories, session-e2e soak, canaries, hygiene)
ci: add the nightly test tier
ci: add the nightly test tier (#720)
ci: remove ci.yml's redundant workspace test job (#755)
ci(release): verify-ci gate, dry_run, --locked, arm64 recollapse
ci(release): verify-ci gate, dry_run, --locked, arm64 recollapse (#757)
test(minvmd): resolve krun_smoke_child via NEXTEST_BIN_EXE when archived
ci(macos): run the daemon lifecycle proof on the mac testbed
ci(macos): darwin testbed — the mini stops compiling (#779)