libbb: leave vforked children through _exit by tombl · Pull Request #4 · tombl/busybox

tombl · 2026-10-25T11:53:52Z

Every HTTPS wget in the lowland guest exited 139 ("RuntimeError: unreachable"), even when the download succeeded. This reworks the fix into one libbb-level mechanism that covers every vforked child: bb_clone CLONE_VM children, vfork()/xvfork() children, and children in applets our distro config doesn't build yet.

Cause

A child made by vfork(), or by clone(CLONE_VM|CLONE_VFORK), runs in its parent's memory until it execs or exits. Calling exit() there is undefined under POSIX: it runs the parent's atexit handlers and flushes the parent's stdio buffers. Since musl 8cca79a7 ("exit: add back lock to make concurrent calls to exit safe"), musl's exit() also stores the caller's thread id in exit_lock and traps if that id calls exit() again. A vforked child has its parent's thread id, so the parent crashes later in its own exit().

This bug is upstream. Upstream busybox calls xfunc_die() after xvfork() when an exec fails: wget's openssl helper does it, and so does fork_or_rexec → re_exec. The raw vfork() children in reformime, telnetd and others die through *_and_die too. This fork's xvfork → xclone conversion only kept the bug. It reproduces natively, with no wasm involved: on x86_64 with musl 1.2.6 and a static NOMMU build, wget https://127.0.0.1:1/ with no openssl, time /nonexistent and reformime -X /nonexistent each segfault in the parent.

Change

libbb: mark vforked children
- bb_vforked_child() is nonzero in any child that shares its parent's memory.
- It's set by the new bb_vfork() and xvfork() macros, which wrap vfork(), and by bb_clone() for CLONE_VM.
- bb_clone() now rejects CLONE_VM without CLONE_VFORK, since the parent would then run while the child's mark is set.
- The parent's signals stay blocked from just before the vfork until it has undone the mark, so none of its handlers ever runs while the child's mark is set. The child restores the original mask before doing anything else. This closes the gap where a handler escaping with siglongjmp could leave the parent marked.
- The new libbb/vforked_child.c keeps the vforks in progress in each memory as one ordered list, starting at the process that made the first one. Each level holds the signal mask its parent restores and, once the child has started, the child's pid. Each parent is suspended until its child is done, so vforks nest and finish in reverse order. A resuming parent pops back to its own level.
- None of this lives in the calling frame, because the child may reuse those stack slots. Clang does so in telnetd (round 1).
- Pids are compared only as each process sees itself, never through getppid(), which is 0 for a child in a new pid namespace (round 12). A resuming parent finds its level by its own pid, as it recorded it, together with a token. The token is the address of an object in its frame (the bb_vfork macro's local, or bb_clone's arguments at the top of the child's stack). That way, another level's parent with the same pid number in another namespace can't be mistaken for it (round 13). The mark search runs from the newest level down, because the running process is the newest one. A resuming parent copies its mask, then lowers the depth, with a compiler barrier in between (rounds 11 and 12).
- A copy of this memory must not inherit the list. On MMU builds, every fork() and daemon() in busybox now goes through bb_fork() and bb_daemon(), which empty the list in the child (bb_daemon() whenever the pid changed, since glibc's daemon() can fail after forking). They're routed by macros in libbb.h, next to the NOMMU fork()/daemon() macros that already exist there, so they also cover applets our config doesn't build. Private bb_clone() children empty the list themselves. In a glibc defconfig build, the only calls to libc fork/daemon are inside the two wrappers.
- This replaces the pthread_atfork handler from v8/v9. Its availability varies too much (rounds 8-10): glibc before 2.28 and uClibc before 1.0.18 keep it in libpthread, and uClibc NOMMU linuxthreads lacks it. Round 10 tried an MADV_WIPEONFORK page instead, but that needs a fallback for old kernels and NOMMU, and in that fallback a recycled pid/ppid pair could mark a copy. The wrappers have no such fallback.
- A copy made some other way (a raw clone(), of which busybox has none) must call the exported bb_forget_vforks() before running busybox code. Until it does, a recycled pid could make it look vforked (round 12), and its vforks may fail with EAGAIN. libc's own popen/system/posix_spawn children run only libc code before they exec, so they're unaffected.
- Where vfork() is really fork(), as on some uClibc builds, the child counts as vforked and restores its mask from its level. That's harmless: it is code written for a vforked child, and its exit() would flush the stdio buffers it copied from its parent a second time (round 8).
- Pids come from the kernel (syscall(SYS_getpid)), not getpid(). Older glibc (2.19 tested) updates its cached pid across vfork() in memory the parent shares.
- bb_clone() now accepts only 0 or CLONE_VM|CLONE_VFORK. Every caller already uses one of the two.

Scope

Per your call after round 8, nothing is guaranteed once a parent is killed while it is suspended in a vfork. Its own parent then resumes while the killed parent's children still run in the same memory, and possibly on the stack the resumed parent is now reusing. No bookkeeping can make that safe. So this version drops all of v8's orphan machinery: the orphan table and its placeholders, pruning, capacity refusal, and the per-level sequence numbers. Orphans aren't marked, so their exit() can disturb the owner's libc state, just as their malloc or stdio can.

Your remaining requirement still holds without the machinery: the owner and any parent that resumes keep their own mark and signal mask. These things make that true, and they are kept on purpose:

An orphan can't write into a list in use. Orphans aren't in the list, so bb_vfork_prepare() refuses them with EAGAIN while it holds any level. v8 needed an orphan table because it treated every unknown process as an untold copy and reset the list for it. That reset is gone, which is why copies made outside busybox's fork()/daemon()/bb_clone() must now call bb_forget_vforks() themselves. Once the list is empty again, an orphan may start a new one. The owner's own vforks then fail with EAGAIN until the orphan's vfork is done, but its mark and mask are untouched.
A child that starts late can only join the newest level, and only while that level is unclaimed. That puts it above every level that remains, so it never writes a protected process's level. A resuming parent also copies its mask before lowering the depth, so nothing can reuse its level first. bb_clone's arguments stay at the top of the child's stack, so a late child never runs on a dead parent's reused frame.

The round-8 timing probes that broke the owner in v8 (delayed_preclaim, real_vfork_preclaim, orphan_before_parent_finish, resumed_before_parent_finish) now leave the owner and the resumed parent clean. What they still show is that the new sibling after a death may lose its mark or its ability to vfork. That's within the narrowed scope.
2. libbb: leave vforked children through _exit

Every exit() in busybox now goes through bb_exit() (via a macro in libbb.h), which calls _exit() in a vforked child. That covers *_and_die, xfunc_die(), bb_show_usage(), exit_FAILURE(), and anything that calls exit() directly, without changing any call site.
xfunc_die() also skips die_func in a vforked child. die_func longjmps into, or cleans up after, the parent's code: a NOFORK applet's caller, a terminal to restore, a temporary file to remove.

use bb_vfork() for the remaining vfork() calls: inetd, telnetd, init, login, reformime, runsv, runsvdir and svlogd. These are mechanical substitutions that keep vfork()'s return convention.
testsuite: exercise vforked children that fail to exec:
- time.tests checks that a missing program exits 127.
- A new reformime.tests checks that -X with a missing program exits 147, which is reformime's helper status plus 20.
- wget/wget-https-without-openssl-fails-cleanly checks for exit 1. It uses -Y off, so a configured proxy can't bypass the helper, plus -T 5, and connects to port 0, which always refuses.

The generic code has no wasm-specific parts. The comments explain why, so the series could go upstream as-is, but per your instruction I'm not sending it.

Validation

Native x86_64, musl 1.2.6, static NOMMU defconfig:
- Baseline: all three new tests fail (139).
- This branch: all three pass.
- Full testsuite on both the NOMMU and the MMU static builds: no regressions. Every failure also fails on the v8 build in the same environment.
Native x86_64, glibc, MMU defconfig: builds cleanly. That defconfig compiles every converted applet, including the ones our wasm config disables.
Clang 21 -Oz and GCC 16 -Os probes, linked against this libbb: after the child scribbles 4 KiB of locals, the parent's mark and signal mask are intact. The round-1 API failed this, ending with mark 90 and SIGUSR1 blocked. Also passing: nesting to depth 8 then EAGAIN, alternating copy and vfork generations (12 deep, through both bb_clone and fork), pending signals with siglongjmp, SIGCHLD, exit and xfunc_die avoiding parent callbacks, an injected vfork failure, fork-backed vfork() through libc fork() and through a raw SYS_fork (marked, mask restored), a busybox fork() copy of a vforked child (unmarked, can vfork), a raw SYS_fork copy (unmarked, refused with EAGAIN), round 10's recycled pid/ppid pair (unmarked), round 11's owner-mask reuse after a killed middle (the owner copies its mask before giving up its level), glibc daemon() failing with EMFILE after forking (the copy is cleared), a shared child in a new pid namespace (marked, mask restored; reformime -X and time also give 147 and 127 under unshare -Urp), a fork-backed vfork whose parent exits before the child starts, an owner whose pid 1 also appears in a nested namespace when the middle process is killed (round 13), and recycled pids in private copies (unmarked). With a killed middle generation, the owner and the resumed parent keep their own mark and mask, including through the four round-8 timing probes.
glibc defconfig (dynamic, MMU): builds cleanly, and the only calls to libc fork/daemon are the two in bb_fork/bb_daemon.
wasm32 (lowland), via busybox: leave vforked children through _exit distro#203: busybox-check-networking, -processes and -smoke pass on this pin (#203's CI). Those checks include the HTTPS-wget and time regressions, which return 139 on the old pin.
busybox: use ash as /bin/sh distro#198 (ash): all four of its distro/busybox/ patches still apply to this rev. They touch only shell/ and coreutils/timeout.c. Its five busybox checks passed on v8 and weren't rerun on this rev.

Known limits

A vforked child can still flush stdio explicitly before it dies, through fflush_stdout_and_exit or bb_error_msg's fflush_all. Those flushes empty the parent's buffers early but don't duplicate anything.
Returning from main() in a vforked child bypasses the macro, but that was already fatal, because the child would return through its parent's stack frame.

Every HTTPS wget in the guest exited 139 after a successful download. wget's openssl helper is a CLONE_VM|CLONE_VFORK child. With no openssl binary its exec fails and it died through exit(). That left musl's exit_lock holding the tid it shares with wget, so wget's own exit() trapped. The bumped tombl/busybox (tombl/busybox#4) marks every vforked child, whether made by bb_clone, bb_vfork or xvfork. It routes busybox's exit() and xfunc_die() through _exit in such a child. The networking check now fails an HTTPS wget to a closed port and a `time` of a missing program, both of which trapped on the old pin.

A child made by vfork(), or by bb_clone() with CLONE_VM|CLONE_VFORK, runs in its parent's memory until it execs or exits. Anything libc or busybox does there on the child's behalf happens to the parent. Add bb_vforked_child(), which is nonzero in such a child, so that code which would disturb the parent's state can tell. A child is vforked if it was made by the new bb_vfork() or xvfork() macros, which replace bare vfork(), or by bb_clone() with CLONE_VM. bb_clone() now accepts only 0 or CLONE_VM|CLONE_VFORK, because without CLONE_VFORK the parent would run alongside its child. libbb/vforked_child.c keeps the vforks in progress in each memory as one ordered list, starting at the process that made the first one. Each parent is suspended until its child is done, so vforks nest and finish in reverse order. Each level holds the signal mask its parent restores and, once the child has started, the child's pid. A resuming parent finds its level by its pid as it sees itself and by a token, the address of an object in its own frame: in a new pid namespace, a pid alone can repeat. The list lives there rather than in the calling frame, because the child runs on that frame and the compiler may reuse its slots for the child's own locals; clang does so in telnetd. Pids come from the kernel, not getpid(), because older glibc updates its cached pid across vfork() in memory the parent shares. A copy of this memory must not inherit the list. On MMU builds, every fork() and daemon() in busybox now goes through bb_fork() and bb_daemon() (via macros in libbb.h, like NOMMU's existing fork() macro), which empty the list in the child, and so do private bb_clone() children. A copy made some other way must call bb_forget_vforks() before it runs busybox code; busybox makes none. Where vfork() is really fork(), as on some uClibc builds, the child counts as vforked, which is harmless: it is code written for a vforked child. The parent's signals stay blocked from just before the vfork until it resumes, so none of its handlers runs in the window. The child restores the original mask before it does anything else. Nothing is guaranteed once a parent is killed while it is suspended in a vfork: its children keep running in the same memory, and possibly on the stack that its own parent resumes on. The list only sees to it that such children can't change the levels of the processes that remain.

A vforked child that exits through exit() runs its parent's atexit handlers and flushes the parent's stdio buffers, which POSIX leaves undefined. On musl the damage outlives the child. Since musl commit 8cca79a7 ("exit: add back lock to make concurrent calls to exit safe"), exit() stores the caller's thread id in a lock and traps if that id calls exit() again. A vforked child shares its parent's thread id and memory, so its exit() leaves the lock held, and the parent then crashes in its own exit(). wget hits this whenever it fetches an https URL on NOMMU without an openssl binary installed. Its helper child fails to exec and dies through xfunc_die(). wget falls back to its own TLS code, completes the download, and segfaults on exit. Any vforked child that fails to exec and dies through *_and_die does the same: re_exec(), BB_EXECVP_or_die() in time, reformime and others. Route every exit() through a new bb_exit(), which calls _exit() in a vforked child, so no call site has to remember to. xfunc_die() also skips die_func in such a child. die_func longjmps into, or cleans up after, the parent's code: a NOFORK applet's caller, a terminal to restore, a temporary file to remove.

inetd, telnetd, init, login, reformime, runsv, runsvdir and svlogd still called vfork() directly, so their children were never marked as vforked. Their fatal paths (BB_EXECVP_or_die in reformime, xopen of the pty in telnetd) still ran exit() in the parent's memory. bb_vfork() keeps vfork()'s return convention, so the retry loops in runsv and svlogd are unchanged.

time, reformime and wget each start a helper in a vforked child. On musl each crashed in its own exit() when the helper could not be executed. Check that each now exits with its documented status.

Every HTTPS wget in the guest exited 139 after a successful download. wget's openssl helper is a CLONE_VM|CLONE_VFORK child. With no openssl binary its exec fails and it died through exit(). That left musl's exit_lock holding the tid it shares with wget, so wget's own exit() trapped. The bumped tombl/busybox (tombl/busybox#4) marks every vforked child, whether made by bb_clone, bb_vfork or xvfork. It routes busybox's exit() and xfunc_die() through _exit in such a child. The networking check now fails an HTTPS wget to a closed port and a `time` of a missing program, both of which trapped on the old pin.

busybox is a fork, so its changes live on tombl/busybox rather than in patch files here. The four patches become the wasm-ash branch (tombl/busybox#5), stacked on vfork-child-exit (tombl/busybox#4).

Every HTTPS wget in the guest exited 139 after a successful download. wget's openssl helper is a CLONE_VM|CLONE_VFORK child. With no openssl binary its exec fails and it died through exit(). That left musl's exit_lock holding the tid it shares with wget, so wget's own exit() trapped. The bumped tombl/busybox (tombl/busybox#4) marks every vforked child, whether made by bb_clone, bb_vfork or xvfork. It routes busybox's exit() and xfunc_die() through _exit in such a child. The networking check now fails an HTTPS wget to a closed port and a `time` of a missing program, both of which trapped on the old pin.

busybox is a fork, so its changes live on tombl/busybox rather than in patch files here. The four patches become the wasm-ash branch (tombl/busybox#5), stacked on vfork-child-exit (tombl/busybox#4).

tombl mentioned this pull request Oct 25, 2026

busybox: leave vforked children through _exit tombl/distro#203

Open

tombl force-pushed the vfork-child-exit branch from 33b974a to ef34036 Compare October 25, 2026 14:28

tombl changed the title ~~libbb: leave CLONE_VM children through _exit when they die~~ libbb: leave vforked children through _exit Oct 25, 2026

tombl force-pushed the vfork-child-exit branch from ef34036 to 573c5c0 Compare October 25, 2026 14:51

tombl force-pushed the vfork-child-exit branch from 573c5c0 to db3a936 Compare October 25, 2026 15:15

tombl force-pushed the vfork-child-exit branch from db3a936 to 9086ddb Compare October 25, 2026 15:50

tombl force-pushed the vfork-child-exit branch from 9086ddb to 0fd50ac Compare October 25, 2026 17:49

tombl force-pushed the vfork-child-exit branch from 0fd50ac to 33039b7 Compare October 25, 2026 18:05

tombl force-pushed the vfork-child-exit branch from 33039b7 to 8da76d9 Compare October 25, 2026 19:05

tombl force-pushed the vfork-child-exit branch from 8da76d9 to e142fc8 Compare October 25, 2026 19:27

tombl force-pushed the vfork-child-exit branch from e142fc8 to 59df6ea Compare October 25, 2026 20:47

tombl force-pushed the vfork-child-exit branch from 59df6ea to 1b96d83 Compare October 25, 2026 20:59

tombl force-pushed the vfork-child-exit branch from 1b96d83 to 7990c43 Compare October 25, 2026 21:13

tombl force-pushed the vfork-child-exit branch from 7990c43 to ce803b9 Compare October 25, 2026 21:23

tombl force-pushed the vfork-child-exit branch from ce803b9 to 8d22716 Compare October 25, 2026 21:36

tombl added 2 commits October 26, 2026 05:46

testsuite: exercise vforked children that fail to exec

0ef30c9

time, reformime and wget each start a helper in a vforked child. On musl each crashed in its own exit() when the helper could not be executed. Check that each now exits with its documented status.

tombl force-pushed the vfork-child-exit branch from 8d22716 to 0ef30c9 Compare October 25, 2026 21:46

tombl mentioned this pull request Oct 27, 2026

shell/ash: run ash on wasm through callback clone #5

Open

tombl mentioned this pull request Oct 27, 2026

busybox: use ash as /bin/sh tombl/distro#198

Open

Navigation Menu

Repository navigation

libbb: leave vforked children through _exit - #4

libbb: leave vforked children through _exit#4
tombl wants to merge 4 commits into
masterfrom
vfork-child-exit

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

Uh oh!

Cause

Change

Scope

Validation

Known limits

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant