norrietaylor · Jul 26, 2026 · Jul 21, 2026 · Jul 22, 2026 · Jul 22, 2026 · Jul 22, 2026
@@ -0,0 +1,179 @@
+---
+title: Release pipeline
+description: How releases are built, staged, smoke-tested, promoted, and installed: release.yml, nightly.yml, promote.yml, prune-releases.yml, and the curl|sh installer.
+---
+
+> This is internal documentation. It is not published to the docs site.
+
+# Release pipeline
+
+Distribution is bucket-centric. A release run stages an **immutable**
+`gs://minimal-one/versions/<short-sha>/` folder (artifacts + a `components`
+manifest); which version users actually receive is decided separately by
+mutable **channel pointer files** (`stable`, `unstable`, `nightly`) at the
+bucket root, written by [`scripts/set-channel.sh`](../../scripts/set-channel.sh).
+Staging is inert; flipping a pointer is the single, cheap, reversible action
+that ships (or rolls back) a version. All GCS writes authenticate via GitHub
+OIDC / Workload Identity Federation.
+
+## release.yml: build, sign, stage
+
+[`.github/workflows/release.yml`](../../.github/workflows/release.yml) runs on
+manual `workflow_dispatch` (inputs: `dry_run`, `skip_ci_verify`), via
+`workflow_call` from nightly.yml, or on a `vMAJOR.MINOR.PATCH[-suffix]` tag
+push.
+
+**verify-ci gate.** The `verify-ci` job requires the five lane aggregators,
+`ci-success`, `ci-linux-native-success`, `ci-linux-kvm-success`,
+`ci-macos-success`, `ci-shell-installer-success` (the required checks on
+`main`; see [docs/ci-strategy.md](../ci-strategy.md)), to have reported
+success on the exact commit being released. `skip_ci_verify` is an admin
+override for green-but-unreported commits.
+
+**Build jobs.**
+
+- `build-release-linux-{amd64,arm64}`: static musl builds of `mip`, `min`
+  (package `minimal`), and `minimald`, one cargo invocation per package so the
+  fat-LTO links serialize. The amd64 job additionally builds a native-glibc
+  `minvmd` against a materialized libkrun prefix (there is currently no
+  `minvmd-linux-arm64` release artifact).
+- `build-release-macos-arm64` (self-hosted Apple Silicon, gated on the
+  `RUN_MACOS_CI` kill-switch): builds `minvmd` (libkrun /
+  Hypervisor.framework) and `min`, rewrites minvmd's libkrun linkage to
+  `@rpath` ([`scripts/rewrite-macos-linkage.sh`](../../scripts/rewrite-macos-linkage.sh)),
+  verifies `min` links only system libraries, and Developer-ID-signs both
+  (hardened runtime + timestamp, notarization-ready).
+- `build-libkrun-macos-arm64`: builds the trimmed `libkrun.1.dylib` shipped to
+  macOS users (same pinned build every macOS CI lane tests against) on a
+  hosted runner; `sign-macos-artifacts` re-signs it and the darwin `gvproxy`
+  with the Developer ID identity on the self-hosted runner.
+- `fetch-release-guest-artifacts`: guest kernel Image and ext4 rootfs (cache
+  pulls keyed by the pinned upstream commit) plus pin-verified `gvproxy`
+  binaries, for both guest arches.
+- `build-release-initramfs`: packs the guest initramfs (minimald as pid-1)
+  from the same shipped musl `minimald` binaries.
+- `compute-version-string`: reads the version out of a built binary so the
+  GitHub Release name matches `minimald -V` exactly.
+
+**release job.** Downloads everything, generates shell completions for `mip`,
+`min`, and `minimald`, then:
+
+- uploads a legacy `minimalone-<sha>.tar.zst` bundle to
+  `gs://minimal-shim/archives/`, retained for backward compatibility with
+  the legacy `minimal-shim` archive path;
+- creates the GitHub Release (tag `release-<sha>`, or the pushed `v*` tag; a
+  suffixed tag becomes a prerelease) with all binaries, guest artifacts, and
+  `completions.tar.gz`.
+
+**stage-installer job** (skipped entirely on `dry_run`):
+
+- [`scripts/stage-release.sh`](../../scripts/stage-release.sh) uploads the
+  artifacts and a `components` manifest (one row per component/os/arch with
+  its SHA-256, kind, and install destination: the authoritative per-platform
+  component list) to immutable `gs://minimal-one/versions/<short-sha>/`;
+- a version-pinned copy of [`scripts/install.sh`](../../scripts/install.sh)
+  is uploaded next to them, so each version's install path is self-contained;
+- [`scripts/set-channel.sh`](../../scripts/set-channel.sh) points the
+  `unstable` channel at the new version. `unstable` auto-advances on every
+  release: no gate.
+
+## nightly.yml: daily cut + smoke + nightly channel
+
+[`.github/workflows/nightly.yml`](../../.github/workflows/nightly.yml) runs at
+10:00 UTC. A `check` job skips the rebuild when HEAD is already staged (public
+HEAD request on the version's `components` manifest), then invokes release.yml
+via `workflow_call`. Three smoke jobs run the shared session e2e
+([`scripts/session-e2e.sh`](../../scripts/session-e2e.sh)) against the
+**shipped** artifacts (native Linux daemon, Linux + KVM microVM, and the
+signed macOS binaries assembled in the installer layout), feeding a
+skip-tolerant `smoke-success` aggregator. Only then does `promote-nightly`
+flip the `nightly` pointer via set-channel.sh. `nightly-tests.yml` is the
+separate 06:00 UTC nightly *test* tier, unrelated to releasing; see
+[docs/ci-strategy.md](../ci-strategy.md).
+
+## promote.yml: gated promotion to stable
+
+[`.github/workflows/promote.yml`](../../.github/workflows/promote.yml)
+(workflow name `promote-cli`) is the manual path that moves the `stable` (or
+`unstable`) pointer. Inputs: `sha` (defaults to the latest staged version in
+the bucket), `target`, `dry_run`, `override_provenance`. The `gate` job opens a
+promotion-approval GitHub issue, notifies Slack, and polls until someone on the
+workflow's approver allowlist (the initiating user is excluded) comments
+`approved` / `denied`; closing the issue without approval counts as denial.
+
+**nightly-provenance gate.** Before flipping the pointer, the `promote` job
+runs [`scripts/verify-nightly-provenance.sh --sha "$SHA"`](../../scripts/verify-nightly-provenance.sh),
+which queries GitHub Actions to confirm the version was built by a successful
+`nightly.yml` run whose Linux smoke jobs actually ran and passed (the
+skip-tolerant `smoke-success` aggregator alone is not trusted, so a no-op
+nightly run that skipped its smokes cannot vouch for a SHA; `smoke-macos` may
+be skipped by the `RUN_MACOS_CI` kill-switch). This rejects manually
+dispatched release.yml runs and tag-push builds. Staging a version does not
+make it promotable. The `override_provenance`
+emergency input bypasses this gate when checked; use it only for documented
+emergency releases that skip the nightly path.
+
+The `promote` job then runs set-channel.sh, which verifies the version is
+actually staged before flipping the pointer. Finally it dispatches a
+`reference-docs-promoted` repository event that triggers a rebuild of the
+published reference docs at docs.minimal.dev.
+
+## prune-releases.yml: GitHub Release housekeeping
+
+[`.github/workflows/prune-releases.yml`](../../.github/workflows/prune-releases.yml)
+runs at 04:17 UTC on Mondays and Fridays (plus manual dispatch, which defaults
+to a dry-run preview) and deletes auto-cut `release-<sha>` GitHub Releases and
+their tags once they age out (default: older than 2 months, always keeping the
+10 newest). The logic lives in
+[`scripts/prune-releases.sh`](../../scripts/prune-releases.sh); exact
+`vMAJOR.MINOR.PATCH` releases are never deleted (hard guard), and the
+installer serves from GCS, so pruning removes redundant copies, not live
+install targets.
+
+## install.sh: channel → version → components
+
+[`scripts/install.sh`](../../scripts/install.sh) is the strict-POSIX `curl |
+sh` installer. Against `https://storage.googleapis.com/minimal-one` it fetches
+the channel pointer (default `stable`; the per-channel endpoints under
+`go.minimal.dev/<channel>` serve this same script with a
+`MINIMAL_INSTALL_TARGET_OVERRIDE=<channel>` line injected to pin the target),
+resolves the version, fetches the
+immutable `versions/<version>/components` manifest (refusing an unknown
+`# format:` header), and for each row matching the host os/arch downloads,
+SHA-256-verifies, and atomically installs the file. The on-disk hash is the
+skip oracle, so reruns only touch changed components, and a running daemon is
+stopped before an executable is swapped. Per-platform sets (from
+stage-release.sh's `COMPONENTS` table): Linux amd64/arm64 get `bin/min`,
+`bin/mip`, `bin/minimald`, a `git-remote-min` symlink, and the AppArmor
+profile/tunable/loader under `data/`; macOS arm64 gets `bin/min`,
+`bin/minvmd`, `bin/gvproxy`, `lib/libkrun.1.dylib`, the `git-remote-min`
+symlink, and the guest payload (`data/{vmlinuz,rootfs.img,initramfs.cpio}`).
+It also wires shell integration (PATH init files, `min` completions, one
+marker-fenced rc block). `--uninstall` reverses all of it offline from the
+local install record, keeping user-modified files unless `--force`;
+`--purge` also removes the data/state/cache trees.
+
+## Runbook
+
+**Cut a release.**
+
+1. Pick a `main` commit whose five lane aggregators are green.
+2. Actions → *Release* → Run workflow (use `dry_run` to rehearse), or push a
+   `vX.Y.Z` tag for a versioned GitHub Release.
+3. The run stages `gs://minimal-one/versions/<short-sha>/` and auto-points
+   `unstable` at it. Nightly runs do the same for `nightly` after smoke.
+
+**Promote to stable.**
+
+1. **Verify the SHA was nightly-built.** The promote workflow rejects versions
+   not built and smoke-tested by `nightly.yml`: manually dispatched or
+   tag-staged SHAs fail the provenance gate after the multi-hour approval
+   completes, even if a later no-op nightly run carries the same SHA. If you
+   need to promote a non-nightly version in an emergency, check
+   `override_provenance` when dispatching.
+2. Actions → *promote-cli* → Run workflow with `target: stable` (optionally a
+   specific staged `sha`; empty means the latest staged).
+3. A second person from the approver allowlist comments `approved` on the
+   auto-opened issue (the initiator cannot self-approve).
+4. The provenance check runs, then set-channel.sh flips the `stable` pointer.
+   Rollback is the same workflow pointed at a previously staged version.
@@ -5,12 +5,17 @@
 # Promotion (promote.yml) only lets a channel point at a version the nightly
 # release workflow actually produced. A version passes when the GitHub Actions
 # API shows a successful run of the nightly workflow whose head commit matches
-# the staged version's short SHA; otherwise this script exits non-zero and the
-# promotion fails before the channel pointer is touched.
+# the staged version's short SHA AND whose smoke-success aggregator job actually
+# ran (conclusion "success", not "skipped"); otherwise this script exits
+# non-zero and the promotion fails before the channel pointer is touched.
 #
-# A nightly run that skipped its build because the SHA was already staged still
-# counts: the run pointed the `nightly` channel at that SHA, so the version went
-# through the nightly path either way.
+# The smoke-success aggregator is skip-tolerant by design (a no-op night or a
+# kill-switch skip still concludes "success"), so this script additionally
+# requires the Linux smoke jobs themselves to have run and passed. A no-op
+# nightly run that skipped its build (the SHA was already staged) skips those
+# jobs and does NOT pass verification — the version must have been smoked by a
+# run that actually built and tested it. smoke-macos alone may be skipped (the
+# RUN_MACOS_CI kill-switch).
 #
 # Usage:
 #   scripts/verify-nightly-provenance.sh --sha SHORTSHA [options]
@@ -72,18 +77,59 @@ command -v gh >/dev/null 2>&1 || die "gh not found"
 
 sha_lc=$(printf '%s' "$SHA" | tr '[:upper:]' '[:lower:]')
 
-# One "head_sha html_url" line per successful run. Capture before grepping so a
-# grep -m1 early exit can't SIGPIPE gh under pipefail.
+# One "head_sha run_id html_url" line per successful run. Capture before
+# grepping so a grep -m1 early exit can't SIGPIPE gh under pipefail.
 runs=$(gh api --paginate \
     "repos/${REPO}/actions/workflows/${WORKFLOW_FILE}/runs?status=success&per_page=100" \
-    --jq '.workflow_runs[] | "\(.head_sha) \(.html_url)"')
+    --jq '.workflow_runs[] | "\(.head_sha) \(.id) \(.html_url)"')
 
 # Anchor the prefix match to the head_sha field (hex run, then the space
-# before the URL) so the input can only ever match a prefix of the commit SHA.
+# before the run_id) so the input can only ever match a prefix of the commit SHA.
 match=$(printf '%s\n' "$runs" | grep -m1 -E "^${sha_lc}[0-9a-f]* " || true)
 
 [ -n "$match" ] || die "version '$SHA' was not built by a successful ${WORKFLOW_FILE} run in ${REPO}; \
 promote a nightly-built version, or re-dispatch with the emergency provenance override"
 
-printf 'verify-nightly-provenance: %s was built by %s run %s\n' \
-    "$SHA" "$WORKFLOW_FILE" "${match#* }" >&2
+# Extract run_id (second field) and html_url (third field onwards).
+run_id="${match#* }"      # drop head_sha
+run_id="${run_id%% *}"    # keep only run_id
+run_url="${match#* }"     # drop head_sha
+run_url="${run_url#* }"   # drop run_id, leaving html_url
+
+# Verify that smoke tests actually ran and passed. The smoke-success aggregator
+# is `if: always()` and skip-tolerant: on a no-op nightly run (the SHA was
+# already staged, so build and smokes were all skipped) it still concludes
+# "success". Checking it alone would let an unsmoked version through, so the
+# Linux smoke jobs are checked individually below; smoke-success is still
+# required to catch failures and cancellations the aggregator does red on.
+smoke_jobs=$(gh api \
+    "repos/${REPO}/actions/runs/${run_id}/jobs?per_page=100" \
+    --jq '.jobs[] | select(.name == "smoke-success" or .name == "smoke-linux-amd64" or .name == "smoke-linux-kvm") | "\(.name) \(.conclusion)"' 2>/dev/null || true)
+
+smoke_conclusion=$(printf '%s\n' "$smoke_jobs" | awk '$1 == "smoke-success" { print $2; exit }')
+
+if [ "$smoke_conclusion" != "success" ]; then
+    if [ -z "$smoke_conclusion" ]; then
+        die "version '$SHA' in ${WORKFLOW_FILE} run ${run_url} has no smoke-success job; \
+smoke tests must complete before promotion"
+    else
+        die "version '$SHA' in ${WORKFLOW_FILE} run ${run_url} has smoke-success with conclusion '$smoke_conclusion' (not 'success'); \
+smoke tests must pass before promotion"
+    fi
+fi
+
+# The aggregator passed; now require the Linux smoke jobs to have actually run
+# and succeeded, so a no-op night (all smokes skipped) cannot wrap an unsmoked
+# SHA into a promotable run. smoke-macos is deliberately not required: the
+# RUN_MACOS_CI kill-switch may skip it, and smoke-success already reds on its
+# genuine failures.
+for job in smoke-linux-amd64 smoke-linux-kvm; do
+    conclusion=$(printf '%s\n' "$smoke_jobs" | awk -v j="$job" '$1 == j { print $2; exit }')
+    if [ "$conclusion" != "success" ]; then
+        die "version '$SHA' in ${WORKFLOW_FILE} run ${run_url} has ${job} with conclusion '${conclusion:-absent}' (not 'success'); \
+the smoke jobs must actually run and pass before promotion (a no-op nightly run does not count)"
+    fi
+done
+
+printf 'verify-nightly-provenance: %s was built by %s run %s (smoke-success: %s, linux smokes: success)\n' \
+    "$SHA" "$WORKFLOW_FILE" "$run_url" "$smoke_conclusion" >&2