Repository navigation
T61 — Two named VMs on one machine, end to end - #1850
Conversation
NET-052..NET-059 end to end, as a VM-backed case: stop the default VM first so both boots (and their `starting VM` log records) come from this case, activate one box per VM, then prove: - the second VM owns its own state dir, bridge socket and daemon while every default-VM path stays put at the provider root (NET-052..054) - one `min ls` attributes each box row to its VM, text and JSON alike (NET-057) - a flagless `min session attach` by box name resolves and hands off onto the VM that owns the box (NET-058) - each VM's published hostname port routes its own box and refuses the other's, at the same time (NET-059) - stopping the named VM leaves the default VM listed, running and routing, with the named VM's state file still on disk (NET-055) Diagnostics print each daemon's start record from the shared minvmd log, naming its VM and state directory; the case degrades to a plain SKIPPED line off the VM lanes (no guest images, switch or KVM). `min net expose` has not landed, so the exposing half drives `min net forward` by box name and states the remaining gap in its own output line.
…vms-on-one-machine-end-to-end
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configuration
You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
|
Approved: no findings at Nothing to do. Verdict APPROVE, posted as a comment: the App cannot review its own pull request; the check and the label carry it. Clobber: none: |
The lane that runs the two_named_vms_on_one_machine case failed in CI on "the session has no socat at /usr/bin/socat" — a line two different failures share: an exec that never reached the box, and a box whose mint came up short. Keep the house message, then print the probe's own stderr, the session's ls row and a peek at the box's /usr/bin, so a recurrence says which one it was. The comment beside it records why a half-materialized box cannot be a race: activate prints an id only once the session is Active. Also correct a comment that claimed the ls table loses its VM column when only one VM is left; every row prints it (crates/minimal/src/cmd/list.rs), so the default's row is asserted by its id alone.
…vms-on-one-machine-end-to-end
|
Round 1 of 10 on a failed check suite, pushed as
Verify scripts/session-e2e.sh runs the two_named_vms_on_one_machine case: on a VM lane it creates a second named VM beside the default one and asserts each VM's own state dir and daemon, one flagless 'min ls' attributing both boxes to their VM, a 'min session attach' by box name landing in the named VM and saying which VM it landed in, both VMs' published proxy ports routing their own box's name and refusing the other VM's at the same time, and the default VM still listed, running and routing after the named one stops; off those lanes it prints its one-line skip, which is what it prints here and on the native CI lane. Nothing in this round changed those beats: the round's commit makes the case's one ambiguous failure line (the socat prerequisite) name which of its two stories it is, and corrects a comment that misdescribed the ls table's VM column. Notes from the agent: THE TWO FAILED CHECKS ARE NOT THIS BRANCH'S CHANGE, AND I LEFT THEM ALONE ON PURPOSE. (1) native-daemon-e2e (and its aggregator ci-linux-native-success) failed on the run's own annotation '::error::the port holder never took 127.0.0.1:7654 — the story cannot be staged', i.e. scripts/session-e2e.sh:4390 inside proof_hostnames_recover_and_two_daemons_route — a proof this branch does not touch (the branch's whole diff in this file is the two_named_vms_on_one_machine case plus its teardown/dispatch entries; the failing step is a bare python3 socket holder, no minimal binary involved). The case cannot have run there: min_daemon is minimald on that lane and the case returns at its first gate. The same bytes are green on main both sides of the failure — run 37093153076 (658187e, this branch's own merge base, finished 03:34) and run 37094358620 (6ede29d, started 03:47) — and the one commit main gained between them (#1593, rename-to-self) does not touch the lane. Verdict: a lane flake, already seen in other shapes on other agent branches (five distinct native-daemon-e2e messages across the recent fai…[truncated] |
The NET-055 section claimed its after-stop `min ls` check proves "a stopped VM contributes nothing", but the named VM's box was destroyed before the stop, so the absence assertion could only ever observe the destroy — a stopped VM with no box contributes nothing by definition, not by behavior. Stop the named VM with its box still live (`minvmd stop` force-drains the guest before it signals the VMM, so a live box is no obstruction), baseline the pre-stop listing on both of the VM's contributions (its box's row, its proxy's discovery line), and assert after the stop that the listing exits 0, prints no skip warning, and shows neither. The default VM's row, daemon and routing beats are unchanged.
…vms-on-one-machine-end-to-end
|
Round 2 of 10 on the runtime's review findings, pushed as
Verify Review threads the coder judged:
The two_named_vms_on_one_machine case in scripts/session-e2e.sh proves the two-named-VM story end to end on a KVM host — and this round makes its NET-055 beat mean what it says: the named VM is stopped while its box is still live, and the one Notes from the agent: The case body cannot run on this box (no /dev/kvm, no guest images), so the test line's pass is a gate-skip, not a body run. Local verification beyond it: bash -n and shellcheck are clean on the script, and a synthetic-listing dry-run replayed the new beats verbatim (format_ls_across_vms shapes) — happy path fires nothing, and each regression mode (stopped VM's box still listed; listing warns about the stopped VM; discovery line survives; default VM's row lost; pre-stop baseline missing the box) fires exactly its own assertion. One deliberate coverage change a reviewer should know: the case no longer runs |
The KVM lane has failed the same beat three rounds running: the two-VM case's first socat probe, whose exec is also the box's MINT — a box mints at its first exec (crates/minimald/src/exec.rs services the request by ensuring the host), and this case is the only one in the lane whose exec runs against a freshly booted VM whose guest store is still empty. The shared session's mint warmed the default VM early in the lane; the named VM's guest has to redo the whole cold fetch of the pkgs universe beside a VM already holding the machine — work no other lane run has ever done. So the probe now retries its exec, bounded, in the shape the harness already gives a fresh activate's first exec (the policy gate's exec retry). A failed fetch leaves nothing sticky behind — the checkout clone lands in a temp dir first (crates/checkouts) — so a retry is a window, not a loosened verdict: socat must still answer before any beat that needs it runs. The probe's error now also names its box and its VM, and says which of its two stories it saw: an exec that never reached the box (its stderr quoted) reads differently from a box that answered without socat. Round 2's diagnostics printed the same evidence into the step log, which is not readable from the check-run API — as an annotation it is.
…vms-on-one-machine-end-to-end
|
Round 3 of 10 on a failed check suite, pushed as
Verify The lane's e2e script still stages the whole two-named-VMs story (per-VM state dirs, one ls with both VMs attributed, routing through each VM's published port, attach resolving the box name to VM alpha, stopping one VM leaving the other), and the beat that has been failing in CI — the first socat probe, whose exec is the box's mint — now retries bounded (the cold second VM's mint fetch re-runs per mint and a failed clone leaves nothing sticky) and its error names the box, the VM and which of the probe's two stories it saw. The case has not yet been carried past that beat by a CI run, so the end-to-end proof (both VMs' names route, stopping one leaves the other) is still owed by the KVM lane; nothing on this box can run it. Notes from the agent: The two failed checks are one failure: test-kvm's whole-lane session-e2e step, which the ci-linux-kvm-success aggregator merely propagates; every other step and every other lane at this head is green. The step's one error annotation is the two-VM case's socat probe, at the same step-log line in rounds 1-3, and the round-2 probe diagnostics (~37 lines) plus the fail() bundle printed into a step log that the check-run annotations API does not expose — raw logs and artifacts need admin rights (403/401), so the probe now re-emits its story as ::error:: lines so the next run's annotations carry the box, the VM and the probe stderr. The macOS e2e (same case, no /dev/kvm gate) is no counterexample: its self-hosted runner never acquires the job ('The job was not acquired by Runner of type self-hosted even after multiple attempts') — rounds 1 and 2 were cancelled that way and this round's is queued to die the same way. Why a retry at all: the probe's exec is also the box's MINT (minimald mints at first exec), the lane's shared session warms the default VM early, and the named VM's guest is the only cold one — the first cold mint-through-a-second-VM any CI lane has run; the pkgs fetch runs inside the mint's graph load and …[truncated] |
The KVM lane's red run refused the named VM's first exec at the guest's SSH host key: "No ED25519 host key is known for alpha". The probe read that as a mint still settling and retried, and its error line carried ssh's message without the fact that decides it. An ssh host-key refusal is a transport fact, not a box fact, and no retry window can produce the record it reads: the VM's own host daemon writes it as the guest reaches READY, before the socket ever serves. So the probe stops retrying on one, and its failure dump names both sides of the pin — the alias ssh asked for (the socket's parent directory: the provider dir's basename for the default VM, the VM's own name for a named one) and the host the record beside that socket answers to, with the record itself printed.
…vms-on-one-machine-end-to-end # Conflicts: # scripts/session-e2e.sh
|
Round 4 of 10 on a failed check suite, pushed as Resolved conflicts in:
Verify Not delivered on the KVM lane: the case is red, and red for a product defect outside this task's layers — minvmd records a named VM's guest SSH host key under the alias 'local-minvmd0' while the client's ssh asks for the VM's own name, so every exec into the named VM's box is refused before the box. A red run is now self-describing: the probe tells that refusal apart from a mint still settling, stops retrying on it, names both aliases in its error line, and dumps the record itself. Notes from the agent: DIAGNOSIS of the failing checks (test-kvm / ci-linux-kvm-success, run 37103214275, the only error annotation the run carried): 'box e2e-two-vm-b on VM alpha: the exec itself failed — the box never answered: No ED25519 host key is known for alpha and you have requested strict checking.' Root cause is in crates/minvmd/src/cmd/mod.rs::read_ready_beacon (called from cmd/boot.rs:144 and cmd/run.rs:636 via default_vm_known_hosts_path, mod.rs:298): it writes the guest's host key into the VM's own provider dir (/providers/local-minvmd0//known_hosts) but keys the ENTRY on paths::provider_instance_name(ProviderKind::Minvmd, 0) — a hardcoded 'local-minvmd0' that ignores the VM name (mod.rs:400/407, with a 'TODO: pass instance_num through'). The client side, crates/minimal-client/src/attach.rs::attach_command (lines 106/188-193), derives both the ssh host alias and the known_hosts file from the SOCKET path: alias = socket's parent dir basename, file = that dir's known_hosts. For the default VM both sides say 'local-minvmd0' (works); for a named VM the socket is /alpha/ssh.sock, so ssh is asked for 'alpha' against /alpha/known_hosts, which exists (strict checking is therefore ON) and holds only the 'local-minvmd0' entry — refused before the box. Consequence: every ssh-riding verb against a named VM's box (session exec, the pty attach, net forward's relay channel, this case's socat probe) is refused, while the RPC verbs work — the case gets thro…[truncated] |
…vms-on-one-machine-end-to-end
The case's two host-side curls could be answered by the host's own proxy settings instead of the surface each exists to probe. The forward's loopback probe handed its request to an external http_proxy on any host that sets one without a 127.0.0.1 exception, failing the proof on the host's settings, not the VM's. two_vm_route was open the other way round: -x pins the proxy, but a NO_PROXY covering the name bypasses even a pinned one, so the probe could read the host's own path, not the VM's. Both now run under the guard bn_page already uses, so this case's answers are the VM's on any host, whatever its proxy env.
…vms-on-one-machine-end-to-end
|
Round 6 of 10 on a failed check suite, pushed as
Verify The case exists and passes its whole NET-052..059 story against a faithful stub of the product's surface; on the real KVM lane it now gets through the state-dir, daemon, one-listing-attribution and published-port beats and then fails at the named VM's first exec with an annotation that names, in one line, the product defect it found — minvmd keys every VM's guest host-key record on 'local-minvmd0' while the client's ssh asks for the VM's own name, so every ssh channel into a named VM is refused at the key — which means the second half of the proof (attach and expose finding the VM from the box name, both VMs' names routing at once, stopping one leaving the other) is blocked by that product defect, outside this case's layer, not by the case. Notes from the agent: Finding (source-confirmed at both ends, reproduced behaviorally, and it is what the red lane has printed since round one — round 5's commit body quotes the run's own message: 'No ED25519 host key is known for alpha'): minvmd writes each VM's guest SSH host key into that VM's own known_hosts (path is VM-aware: providers/local-minvmd0//known_hosts, crates/minvmd/src/cmd/mod.rs default_vm_known_hosts_path) but keys the ENTRY on paths::provider_instance_name(Minvmd, 0) = 'local-minvmd0', hardcoded for every VM (mod.rs:398-411, with a TODO at 406 admitting the key is not plumbed). The client derives the ssh host alias from the bridge socket's parent-dir basename (crates/minimal-client/src/attach.rs:184-193), which is the VM's own name for a named VM, and enforces StrictHostKeyChecking=yes + UserKnownHostsFile= whenever the record exists (attach.rs host_key_opts) — and it always exists: the guest's READY beacon carries the key and the supervisor records it before the socket serves. The two rules agree only for the default VM (its socket dir IS the provider dir), so |
|
The runtime will not revise this pull request again on its own: round 7 would exceed the cap of 6 (the last trigger was a failed check suite). A person decides now. Remove |
|
The runtime will not revise this pull request again on its own: round 7 would exceed the cap of 6 (the last trigger was the runtime's review findings). A person decides now. Remove |
|
The runtime will not revise this pull request again on its own: round 7 would exceed the cap of 6 (the last trigger was a failed check suite). A person decides now. Remove |
…vms-on-one-machine-end-to-end
Best-effort transcript capture in two_named_vms_on_one_machine: each VM's gvproxy forwarder and lease tables, read over its -listen control socket, plus the live gvproxy processes. Captured after the default VM's stop, with both VMs up, after the named VM's stop, and right before the NET-055 dial. It never asserts or fails the case; it is evidence for the KVM lane's dead-guest dial on the default VM's proxy port.
|
I pushed 298dbaf, a diagnostic capture only. It doesn't change any of the case's checks. It comes after merging main (e5adc67).
Each capture is best-effort and never fails the case. The purpose is to confirm, from the KVM lane, the suspected stale-forward defect. The suspicion is that after a VM stops, gvproxy still holds the forward for that VM's port ( |
|
The runtime will not revise this pull request again on its own: round 7 would exceed the cap of 6 (the last trigger was the runtime's review findings). A person decides now. Remove |
|
The runtime will not revise this pull request again on its own: round 7 would exceed the cap of 6 (the last trigger was a failed check suite). A person decides now. Remove |
1 similar comment
|
The runtime will not revise this pull request again on its own: round 7 would exceed the cap of 6 (the last trigger was a failed check suite). A person decides now. Remove |
|
/revise |
…d on On macOS std's SO_REUSEADDR lets the wildcard bind probe succeed over another VM's hostname proxy listening on 127.0.0.1:<port>, so a second VM could be handed a port already served. The TCP probe now binds the published loopback address and then the wildcard, which together refuse a loopback or wildcard holder on macOS and Linux while still tolerating a TIME_WAIT left by a stopped VM.
two_vm_log_since applied the snapshot's line offset to whichever log file was newest at read time. After a daily rotation that offset lands on the new file. It now takes the snapshot path and, on rotation, reads the snapshot file's tail and then the whole newer file.
T61 — Two named VMs on one machine, end to end
slice: S15
kind: integration
covers: none
size: M
est: 60 code, 160 tests
executor: agent
difficulty: routine
model: standard
diagnostics: each VM host daemon log from the run names its VM and state directory
observability: the case prints each command against each VM with its outcome
layers:
- e2e:
- scripts/session-e2e.sh: adds the case two_named_vms_on_one_machine
test: ./scripts/session-e2e.sh two_named_vms_on_one_machine
produces: none
consumes: none
proof: create a second named VM with its own state; ls shows the VM per box; attach and expose find the VM from the box name; both VMs' names route; stopping one leaves the other.
depends on:
- blocked by #1627
- blocked by #1626
- blocked by #1670
Proof
On a VM-capable lane a person can run ./scripts/session-e2e.sh two_named_vms_on_one_machine and watch it create a second named VM with its own state directory, bridge socket and running daemon while every default-VM path stays put at the provider root, then one
min lslisting both VMs' boxes with each row attributed to its VM (text and --json), a flaglessmin session attach e2e-two-vm-bresolving the box name onto the VM that owns it and announcing 'on VM alpha' before proving the box's filesystem from inside, each VM's own published port routing its box's name (200 with its marker) and refusing the other's (502) at the same time, and finally stopping the named VM leaving the default VM still listed, running and routing with the named VM's state file left on disk — with each command printed against its VM and each daemon's 'starting VM' record from the shared minvmd log naming its VM and state directory.Notes from the agent: min net expose has not landed (attach.rs:249-253 notes the landed verb will reuse the same cross-VM resolution, which is already pinned per-path by box_name_resolves_vm_without_flag in crates/minimal). The case therefore proves NET-058's 'attach or min net expose' through the attach half and drives the shipping exposing verb
min net forward e2e-two-vm-a 18093:18090by box name; it prints an honest note in its transcript about the gap and asserts nothing about the forward's cross-VM behavior. The case gates itself to a plain SKIPPED line off the VM lanes (no guest images, no gvproxy, no writable /dev/kvm), which is the path taken on this sandbox; the KVM lane exports everything it needs (MINVMD_KERNEL_PATH/ROOTFS_PATH/INITRAMFS_PATH, MINVMD_GVPROXY_BIN, E2E_VM=1, E2E_MINIMAL_ARGS) and the macOS lane gets them fromjust e2e. The case stops the default VM at its start so both boots and their log records come from this case. The standalone-invocation arm, the teardown named-VM stop, seed-dir cleanup and forward-PID kill were al…[truncated]Tests
./scripts/session-e2e.sh two_named_vms_on_one_machineVerify
min task run agent-verifypassed ([tasks.verify]).Base
The base branch moved while the coder worked and was merged in before the tests ran. These files changed on BOTH sides since the branch point — read them first, because a rewrite from a stale reading can undo a merge without a conflict:
scripts/session-e2e.shCloses #1672
Node-port probe (#1814, partial)
crates/minvmd/src/cmd/run.rssits outside this task's layers. Its NET-059 leg cannot pass on macOS while a second VM can be handed a hostname-proxy port another VM already serves.tcp_bind_probebinds127.0.0.1:<port>, the exact address the proxy publishes on, and then the wildcard. On macOS,SO_REUSEADDRlets a wildcard bind succeed over a loopback listener, but never over a listener on its own exact address, so the two binds together refuse either kind of holder.node_port_bind_probe_refuses_a_port_held_on_loopback.Refs #1814, which stays open: the host-wide port authority (or holding the port until the expose succeeds) and surfacing the cause in
min lsand at session start are still to do.