8000 fix(minimald): reap orphaned zombies when running as pid 1 by norrietaylor · Pull Request #1934 · gominimal/minimal · GitHub
[go: up one dir, main page]

Skip to content

fix(minimald): reap orphaned zombies when running as pid 1 - #1934

Merged
norrietaylor merged 2 commits into
mainfrom
fix/minimald-reap-orphans-pid1
Oct 6, 2026
Merged

norrietaylor merged 2 commits into
mainfrom
fix/minimald-reap-orphans-pid1

Conversation

@norrietaylor
@norrietaylor norrietaylor commented Oct 4, 2026 •
Copy link
Copy Markdown
Member

Problem

In the microVM, minimald runs as pid 1: the kernel runs the initramfs /init, which is this binary (crates/minimald/src/guest.rs module docs; is_microvm_init checks pid == 1 and an argv[0] of init). pid 1 inherits every orphan in the guest. When a session shell outlives its parent, or git detaches a background job, the orphan is reparented to pid 1, and when it exits it stays a zombie until pid 1 waits for it. Nothing in the daemon ever waits for a pid it did not spawn (there is no waitpid(-1) and no SIGCHLD handling in crates/minimald). The guest module even left a note to "revisit if zombie reaping bites". So zombies pile up, one per departed orphan, for the life of the VM. The pid table is finite, so a long-lived VM eventually has to be restarted.

A plain waitpid(-1) loop inside the daemon is not a fix. tokio's process driver reaps tokio::process::Children by pid, and the session host blocks in hakoniwa's Child::wait() on its container's pid (HakoniwaProcess::wait). A wait for "any child" in the same process would take their exit statuses first, so a normal session exit would turn into a hang or a wrong exit code.

Fix

pid 1 now splits in two, the way tini does. At the very top of main, while the process is still single-threaded, reaper::split_init forks:

  • The child runs the daemon, unchanged. Every process the daemon spawns is the daemon's own child, so tokio and hakoniwa reap their own children exactly as on a native host.
  • pid 1 stays behind as a reaper. It only ever blocks in waitpid(-1). Its only children are the daemon and the orphans reparented to it, and an orphan has no other owner, so nothing it reaps belonged to anyone else. If the daemon dies, pid 1 takes the VM down (guest::shut_down_vm), and it never exits itself.

pid-1 specifics:

  • The "am I the microVM daemon" check (guest::is_microvm_daemon, which main's is_minimal_microvm now delegates to) gates reboot(2), the guest boot config, and the classifier's guest refusal. It must survive the fork without becoming spoofable. pid 1 sets a process-local flag in the forked child after passing the existing pid-1 + argv[0] check itself. Nothing outside the guest can set that flag. main.rs's duplicate is_microvm_init and its tests are removed. guest.rs keeps the same function and its spoofing tests.
  • Signals: none are forwarded. The daemon installs no signal handlers, and today, as pid 1, it silentl 8000 y drops every signal sent from inside the guest. Forwarding signals would turn those into ones that kill the daemon.
  • Shutdown: unchanged on the normal path. The daemon still quiesces and calls reboot(2) itself; CAP_SYS_BOOT does not depend on being pid 1. pid 1's shutdown is the backstop for a daemon that dies abnormally. Previously a panicking daemon exited init, which panics the guest kernel.
  • READY marker, vsock, rootfs switch: all happen in the daemon after the fork, so nothing about them changes. pid 1 forks before it opens anything, so it holds no descriptor on the state volume that could block the shutdown unmount.
  • If the fork fails, the daemon runs unsplit as pid 1, which is the previous behaviour.

One trade-off: the daemon is no longer pid 1, so the OOM killer may now pick it. If it does, pid 1 takes the VM down cleanly. Giving it oom_score_adj = -1000 would be inherited by every session process, so this PR leaves that alone.

Alternatives considered

  • A registry of owned pids, with a reaper that peeks via waitid(P_ALL, WEXITED | WNOWAIT) and reaps only unregistered pids. A child can exit between spawn and registration and then be reaped as an orphan. Closing that race means wrapping every spawn site: tokio, hakoniwa, std::process (mkfs, e2fsck), and the nsenter re-exec, plus any spawn added later. That is more code, and it stays fragile.
  • A SIGCHLD handler in the daemon that reaps any child. It steals exit statuses, as described above.
  • An existing init shim in the guest. There isn't one: the initramfs /init is minimald itself, with no /init.krun in between.

The split is the only race-free option: ownership is decided by the process tree, not by bookkeeping.

Tests

reaper::tests::pid_1_reaps_orphans_and_the_daemon_keeps_its_own_children runs the split end to end, with PR_SET_CHILD_SUBREAPER standing in for pid 1:

  • A re-exec of the test binary marks itself a subreaper and runs the real reap loop over a second re-exec that plays the daemon.
  • The daemon half creates an orphan (sh backgrounds a subshell and exits), then checks that a tokio-owned child (exit 7) and a blocking-waited child (exit 5, the way hakoniwa waits) both report their own exit codes while the reap loop runs.
  • The daemon half asserts that the orphan disappears from /proc entirely, zombie included. The reaper half asserts that it reaped exactly that pid.

Each half runs in its own process because a subreaper's waitpid(-1) would race other tests' children in a shared test process.

Verification

  • just ci (macOS, darwin scope): green.
  • just test-cross: cross clippy (-D warnings, Linux-only crates) is clean. The test step stopped at common's shell_harnesses (install_test.sh: id: cannot find name for user ID 502, a container-user issue that this change does not touch) before reaching minimald. So minimald's tests ran with the recipe's own cross command scoped to minimald with --no-fail-fast. The three reaper tests pass, and main.rs's tests pass (test result: ok. 5 passed). The lib suite reports 686 passed; 19 failed. Every failure is outside the changed code: pidfd_open returns ENOSYS under qemu-user, tests that capture tracing output collide when libtest runs them in one shared process (nextest gives each test its own), and emulation pushes some tests past their timeouts.
  • just e2e (macOS, real microVM, with MINIMAL_ALLOW_VERSION_SKEW=1, because a cross-built guest in a git worktree reports a bare 0.7.0): every guest-side proof passes. That covers session exec, rename, task run, the exit-code relay (7 → 7), lifecycle hooks (including surviving a daemon restart), the interactive pty sandbox proof, own-IP egress, and box registration. The run then stops at the host-side native-DNS answerer step, because another minvmd already running on the test host holds 127.0.0.1:7656. The guest diag bundle from that run shows the new shape: pid 1 init (about 1.7 MB RSS) with the daemon as its child (pid 54, ppid 1, in_microvm: true), and no zombies.

Refs gominimal/inbox#555

Summary by CodeRabbit

  • Improvements
    • Improved microVM process management: orphaned processes are reaped while the daemon runs, helping prevent lingering processes.
    • Updated microVM shutdown handling to work with the revised daemon lifecycle.

In the microVM the daemon is pid 1, so every orphan in the guest is
reparented to it, and nothing ever waits for them: each exited orphan
stays a zombie for the life of the VM.

pid 1 now forks the daemon off at startup, before any thread exists,
and stays behind as a reaper that waits for any child. The daemon's own
children stay the daemon's, so tokio's process driver and hakoniwa's
blocking wait keep their exit statuses; pid 1 only ever has the daemon
and orphans as children. When the daemon dies, pid 1 takes the VM down.

The microVM check carries over to the forked daemon through a flag only
pid 1 sets, so it stays unspoofable.

Refs gominimal/inbox#555
@norrietaylor
norrietaylor requested a review from a team as a code owner October 4, 2026 04:51
@coderabbitai
coderabbitai Bot commented Oct 4, 2026 •
Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Essentials
  • Run ID: 0e26f378-9a3d-472a-8904-1f874aa63b74
📥 Commits

Reviewing files that changed from the base of the PR and between d6da92d and 66a6d9e.

📒 Files selected for processing (4)
  • crates/minimald/src/guest.rs
  • crates/minimald/src/lib.rs
  • crates/minimald/src/main.rs
  • crates/minimald/src/reaper.rs

Included review availability: This review used your included allowance. Your plan provides up to 5 included reviews per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

Linux startup now forks the daemon from pid 1 before creating the async runtime. The parent reaps child processes and attempts VM shutdown after the daemon exits. Daemon detection recognizes the forked process.

Changes

MicroVM process reaping

Layer / File(s) Summary
Fork and reap primitives
crates/minimald/src/reaper.rs, crates/minimald/src/lib.rs
Adds the Linux reaper module, a marker for the forked daemon, and wait helpers that distinguish the daemon's exit from other child results.
PID 1 lifecycle and tests
crates/minimald/src/reaper.rs
PID 1 reaps children until the daemon exits, attempts VM shutdown, and continues reaping. Tests check orphan reaping and confirm Tokio and blocking child waits retain their exit statuses.
Startup and daemon detection
crates/minimald/src/main.rs, crates/minimald/src/guest.rs
Linux startup calls split_init before runtime setup. Daemon detection recognizes the forked process or the existing pid-1 and argv[0] match. The local init check and its tests are removed.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant Main
  participant PID1
  participant Daemon
  participant Child
  Main->>PID1: Call split_init before runtime setup
  PID1->>Daemon: Fork daemon
  Daemon->>Daemon: Initialize Tokio runtime
  Child-->>PID1: Exit as orphan
  PID1->>PID1: Reap child with waitpid
  Daemon-->>PID1: Exit
  PID1->>PID1: Attempt VM shutdown and continue reaping
Loading

Suggested reviewers: evanspearman, twitchyliquid64

Merge Risk: ⚪ Minimal · up to 66a6d

No merge-blocking issue was established; the change is ready for normal merge checks.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main change: reaping orphaned processes when minimald runs as pid 1.
Description check ✅ Passed The description explains the problem, fix, alternatives, and test results. It covers the template’s summary and testing requirements, though it does not include a separate Checklist section.
Docstring Coverage ✅ Passed Docstring coverage is 92.86% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 14 functions across 4 files.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

A rabbit watched pid one take its place,
While forks set the daemon on its race.
The orphaned children found their way,
Their exit codes kept safe each day.
Then down the burrow, logs grew quiet.

Comment @coderabbitai help to get the list of available commands.

@minimal-foundry
minimal-foundry Bot commented Oct 4, 2026 •
Copy link
Copy Markdown
Contributor

Reviewed: 1 finding, none blocking at 66a6d9e3dfaa.

Address the comments on the diff.

Clobber: none: git log on all four files shows only this PR's head commit touching them.

@minimal-foundry minimal-foundry Bot left a comment
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minimal-review — advisory review

Verdict COMMENT — 1 finding, 1 inline.

Comment on lines +355 to +358
// As the microVM's daemon we must not return: exiting init panics the
// guest kernel and wedges the VM (#730), and when the daemon runs as pid
// 1's fork, pid 1 would only take the VM down after it, without this
// log line. Take the VM down here instead — a clean
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LOW · style · confidence 0.75 · in shut_down_vm

This PR's whole point is that the daemon is no longer pid 1 — split_init forks it off and pid 1 stays behind as the reaper. Yet guest::shut_down_vm's doc (crates/minimald/src/guest.rs:924) still says "ONLY call this as the microVM's pid-1", and this new comment/block describes the forked daemon calling shut_down_vm as its primary shutdown path. The safety guard (is_minimal_microvm) is correct and the stale doc doesn't change behavior, but the next maintainer reading shut_down_vm will be told the daemon (a fork child, pid != 1) must not call it — and could 'fix' the guard to a pid == 1 check that breaks the normal guest shutdown. The capability comment just below (main.rs:369, "the microVM's pid-1 has it") has the same staleness: the daemon holds CAP_SYS_BOOT via inheritance, not because it is pid 1. The author updated several sibling comments for the new daemon role; these two were missed.

What the reviewer checked

guest.rs:924 "ONLY call this as the microVM's pid-1 (guard with [is_minimal_microvm]): on an ordinary host it reboots the machine." — but the daemon that now calls it (main.rs:367) is pid 1's fork, not pid 1.

Hand this to an agent
Act on a code-review finding.

File:       crates/minimald/src/main.rs
Lines:      355-358
Category:   style
Severity:   low (reviewer confidence 0.75)

Finding:
This PR's whole point is that the daemon is no longer pid 1 — `split_init` forks it off and pid 1 stays behind as the reaper. Yet `guest::shut_down_vm`'s doc (crates/minimald/src/guest.rs:924) still says "ONLY call this as the microVM's pid-1", and this new comment/block describes the forked daemon calling `shut_down_vm` as its primary shutdown path. The safety guard (`is_minimal_microvm`) is correct and the stale doc doesn't change behavior, but the next maintainer reading `shut_down_vm` will be told the daemon (a fork child, pid != 1) must not call it — and could 'fix' the guard to a `pid == 1` check that breaks the normal guest shutdown. The capability comment just below (main.rs:369, "the microVM's pid-1 has it") has the same staleness: the daemon holds CAP_SYS_BOOT via inheritance, not because it is pid 1. The author updated several sibling comments for the new daemon role; these two were missed.

First verify this against the code yourself. It is model output and
may be wrong. If it is wrong, or the tradeoff was deliberate, say so
and change nothing.

👍 useful · 👎 noise — react to tell the reviewer. Replying works too, and says more.

@norrietaylor
norrietaylor added this pull request to the merge queue Oct 6, 2026
Merged via the queue into main with commit 14d8dc1 Oct 6, 2026
34 checks passed
@norrietaylor
norrietaylor deleted the fix/minimald-reap-orphans-pid1 branch October 6, 2026 05:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0