Repository navigation
minvmd: node-port probe hands a second macOS VM a proxy port another VM already holds #1814
Description
Problem
On macOS, minvmd's node-port assignment can give a second VM the hostname-proxy port that a running VM already serves on. That VM's proxy then never comes up.
assign_node_port (crates/minvmd/src/cmd/run.rs:853) checks whether the preferred TCP port is free by binding 0.0.0.0:<port> with std::net::TcpListener. Rust's std sets SO_REUSEADDR on that socket on every non-Windows target. On macOS, a TCP socket with SO_REUSEADDR can bind 0.0.0.0:p while another socket is listening on 127.0.0.1:p. So the probe reports the port as free even though the first VM's published proxy holds 127.0.0.1:7654.
Measured on Darwin 25.6:
- Prober with
SO_REUSEADDR: the wildcard bind succeeds over a127.0.0.1listener, with or withoutSO_REUSEADDRon the holder. - Prober without it: the bind is refused with EADDRINUSE.
- UDP answerer probe (
UdpSocket::bind, which std leaves withoutSO_REUSEADDR): correctly refused.
There is also a race on every OS. The probe socket is dropped before boot, so two VMs booting together can both be handed the same free port.
Impact
- Since #1788 (T63), a VM binds its handed ports pinned and never walks to another port. A duplicated proxy port retries as unavailable for the daemon's lifetime.
- NET-059's proxy half (two VMs' hostname surfaces served at once) then fails on the macOS VM backend.
- #1807 (T60) relies on minvmd handing each VM distinct ports.
Fix options
- Probe
127.0.0.1as well as the wildcard. - Probe with a socket that has no
SO_REUSEADDR, for example throughsocket2. - Hold the probe binding until the guest publishes.
- Keep a per-host ledger of the ports already handed out.
Proof
- Unit test: with a listener held on
127.0.0.1:p,assign_node_port(p, false)returns a port other thanpon macOS and Linux. - VM-lane case: two VMs booted back-to-back get distinct proxy ports, and both serve them.
Activity
norrietaylor commentedon Oct 2, 2026
Design constraint from the architecture review: the fix must arbitrate minvmd's port assignment host-wide, through one authority, the same way design §7.1 arbitrates the reserved range. Probe-then-bind with address-reuse flags is not arbitration. Until this lands, #1807 makes the daemon's proxy bind without
SO_REUSEADDR/SO_REUSEPORT, so a duplicated handed port fails loudly at bind instead of two VMs sharing it.norrietaylor commentedon Oct 2, 2026
Narrowed reuse rule, from the architecture review of #1807. minimald's proxy keeps the platform-default bind (SO_REUSEADDR on Unix, never SO_REUSEPORT). The no-shadow rule belongs to this issue's host-side bind. The fix must:
0.0.0.0, so a wildcard holder and a specific holder can never coexist;fix(minimald): restore the platform-default proxy bind
norrietaylor commentedon Oct 3, 2026
Seen live on T61 (#1850): macOS two-VM case, plus one acceptance criterion from the architecture review
T61's two-VM e2e reproduces this on the macOS lane (run 37111686278). VM
alphais handed 7654 while the default VM serves it. Its guest daemon's gvproxy expose then returnsHTTP 500: listen tcp 127.0.0.1:7654: bind: address already in use, and it logshostname proxy could not publish on the host loopback; retrying with backoff(host_port=7654) for the rest of the run.min lsshows alpha's name surface as "not serving", and NET-059 fails.Fix shape, per the architecture review: allocation stays host-side, in minvmd's
assign_node_port. Probe withoutSO_REUSEADDR(or bind127.0.0.1:p, matching what gvproxy will bind), and hold the probe socket, or reserve the port in minvmd's own table, until gvproxy's expose succeeds. The guest must not relocate, because that would be a second allocator racing minvmd.Added acceptance criterion: a pinned handed port that answers EADDRINUSE must not retry silently for the daemon's lifetime. Address-in-use from another owner is not transient, and the bind discipline makes a failed bind a surfaced error, never a silent fallback. The daemon reports it at session start and in
min ls, naming the port and the cause; it may keep retrying in the background. "not serving" with no cause is not enough.This issue blocks T61 (#1672): until it lands, T61's NET-059 leg on macOS fails because of this defect, not a test bug.
T60 — Address boxes across two VMs by name without a global flag (gom…
norrietaylor commentedon Oct 4, 2026
Remaining scope, after the interim probe lands in #1850. The interim probes
127.0.0.1:<p>, then0.0.0.0:<p>, and releases the port. Two VMs booting at once can still both see the same port free.Correctness (design §7.1 bind discipline: a failed bind is a surfaced error, never a silent fallback). When gvproxy's expose of the node's proxy port fails with
EADDRINUSE, the VM start must do one of two things:It must never come up with no hostname proxy. Today it does exactly that: it warns ("could not publish port … address already in use") and continues, and
min lsshows no HOSTNAME PROXY row for that VM.Race. Hold the port from the probe through the expose, or use a host-wide port authority, so two concurrent boots cannot draw the same port.
Surfacing. The
min lsNAME SURFACE row and the session-start message name the cause, not a bare "not serving".norrietaylor commentedon Oct 4, 2026
The remaining scope (fail with the port named instead of coming up without a hostname proxy, the concurrent-boot race, and naming the cause in
min lsand at session start) is now NET walk task T93, #1949. It has no dependencies, so the walk dispatches it.norrietaylor commentedon Oct 5, 2026
Superseded. The interim node-port probe landed in T61 #1850, and the remaining scope moved to T93 (#1949), which merged as #1957.