vivarium: aarch64-darwin guests; QEMU+HVF cannot answer macOS 13+ PAC hypercalls

Goal: an aarch64-darwin guest in vivarium that runs Nix builds and commands. Spike on macOS 27.0.1 (M5 Pro), 2026-11-03, with a Sonoma tart image and orchard's vmapple tree (github.com/yaelliethy/orchard).

Measured:

QEMU + HVF boots the stock host AVPBooter, then iBootStage1, Sonoma's iBootStage2 and kernel entry. Then the kernel spins on mov x0,#0xc1000000; hvc #0; cbnz x0,..
Cause: public Hypervisor.framework answers SMCCC 0xc1…… (Apple CPU service, the PAC calls) and 0xc6…… in the host kernel with 0xffffffff. QEMU never sees them. PSCI, 0x80…… and 0xc2…… do exit to user space. A probe of each function ID showed this.
hv_vm_create works only in ISA mode 1 for a binary with com.apple.security.hypervisor. Every other mode returns HV_DENIED. Virtualization.framework uses modes 4/1/3 and holds com.apple.private.hypervisor.
So no QEMU with HVF boots macOS 13+. macOS 12 does, but the pinned nixpkgs needs macOS 14 (darwinMinVersion).
QEMU TCG: the stock booter works. Sonoma's iBoot reads unmapped 0x30240040; a stub device fixes that. Then QEMU dies with SIGILL in JIT code (host instruction 0x9ad506d4) at kernel entry, every run.
apple-gfx-mmio crashes on a macOS 27 host (setMapMemory: is gone).

Routes left: a Virtualization.framework backend (Swift launcher, vsock agent, 2 guests per host), or QEMU TCG (debug the SIGILL; slow).

Assisted-By: Claude Opus 5.5

Update: macOS 14.8.7 boots on QEMU + HVF to SSH, and runs commands, about 14 s after boot.

What made it work (orchard tree, local spike change; 165 lines across 5 files):

hvc #0 with an Apple CPU service ID is swallowed by the host kernel, but hvc #1 and smc exit to user space with x0 unchanged (probe).
At the boot CPU's first SET_INITIAL_STATE, QEMU scans RAM and rewrites hvc #0 to hvc #1 where it follows an x0 setup or precedes cbnz x0, . (40 of ~61 sites). The first call itself is caught with a 10 ms vCPU kick and served in place.
HVF data aborts with ISV=0 go through orchard's arm_emulate_ldst (xnu uses pre-index and pair stores on MMIO).
Graphics: gfx=off stub. Network: slirp; cache.nixos.org answers 200 from the guest.

Open: the guest restarts in an orderly way about every 20 s (shutdown_stall reports, no panic). Disabling WindowServer and loginwindow did not stop it. The initiator is not found yet.

Assisted-By: Claude Opus 5.5

Paused. Everything is on bookmark macos-guest-spike (vivarium/docs/design/macos-guest-spike/): the QEMU patch against orchard, probes, scripts, and the notes.

New since the last comment:

Under Virtualization.framework (tart) the same image stays up. So the 22 s reset loop is QEMU's machine model, not the image.
The reset is the guest's PSCI SYSTEM_RESET. launchd logs System shutdown initiated by kernel, so some process calls reboot().
Ruled out: unpatched hypercalls (all hvc #0 rewritten now), GPIO power-key polarity, WindowServer, the NVRAM install phase, the timebase.
Leads: QEMU has PL031 where VZ has avp-rtc; AppleVPKeyStore returns kIOReturnUnsupported; the virtio console, entropy and socket devices are missing.

Next: find the reboot() caller from the shutdown-stall spindump, and stream guest logs to the host so the tail survives the reset.

Assisted-By: Claude Opus 5.5

vivarium: record the macOS guest running Nix builds

Solved the reset loop, and Nix builds run in the guest.

The resets came from tart's root daemon org.cirruslabs.tart-guest-daemon calling reboot(); it wants tart's host over virtio-socket. sudo launchctl disable system/org.cirruslabs.tart-guest-daemon and the guest stays up (141 resets before, none after).
Nix 2.35.4 (official installer, --daemon) in macOS 14.8.7 on QEMU + HVF: a raw derivation builds in 0.7 s, nixpkgs#hello substitutes in 11 s, and --rebuild compiles it from source in 32 s.
A guest shutdown -h powers QEMU off cleanly.

Details and next steps (a darwin QEMU backend for vivarium, carrying the QEMU patch in Nix) are on bookmark macos-guest-spike, vivarium/docs/design/macos-guest-spike/README.md.

Assisted-By: Claude Opus 5.5

vivarium: record virtio-serial and virtio-vsock in the macOS guest

Both agent channels work in the macOS guest on vmapple:

virtio-serial: QEMU's own virtio-serial-pci with a virtserialport named vivarium-agent. macOS binds AppleVirtIOConsole and creates /dev/cu.vivarium-agent. Data passes both ways. No QEMU change.
virtio-vsock: QEMU has no userspace vsock device (vhost needs Linux), so I wrote virtio-vsock-unix-pci, with its host side in unix sockets the way Firecracker maps it. macOS binds AppleVirtIOSocket. 20/20 echo round trips; 64 MiB echoed host to guest at 187 MB/s and guest to host at 121 MB/s, every byte back.

Patches and test tools are on bookmark macos-guest-spike (orchard-vsock.patch, guest/vsockcat.c, host/vsock_connect.py).

Assisted-By: Claude Opus 5.5

vivarium: record the QEMU fork replay of orchard

vivarium: point the macOS guest spike at nixidae/qemu

The QEMU work now lives in a fork, not in orchard's vendored tree: https://github.com/nixidae/qemu/tree/nixidae

Base: upstream f8aef8a9 (2026-09-18). orchard's vendored tree differs from it only in the 27 files orchard changed or added, found by comparing content hashes with upstream commits.
orchard's four QEMU commits are replayed on it with their original author and dates; each message names its orchard commit. Then the vsock and HVF commits.
The result equals orchard's tree plus our changes, file for file. The only extras are upstream files orchard's repo dropped: Rust .wrap files, tests/keys, submodule pointers.

The tools are on bookmark macos-guest-spike: host/find_qemu_base.py, host/replay_orchard.sh.

Assisted-By: Claude Opus 5.5

Newer guests, on nixidae/qemu nixidae (a8a6e107):

The fork now carries two patches from Mohamed Mediouni's qemu-devel series "vmapple: making it work on the latest macOS host releases" v8: apple-gfx on new hosts (05/10) and FileVault off (10/10).
macOS 15.7.7: SSH after 41 s with gfx=on. Without the gfx patch the paravirtual GPU driver panicked on the stub.
macOS 26.6.2: reaches SSH. Without patch 10 it panicked with unencrypted data volume is not allowed.
Both then panic: Undefined kernel instruction instr=d51ef860, an msr to Apple's implementation-defined register S3_6_C15_C8_3, which public HVF refuses. That is the next fix.
macOS 27 (Golden Gate): iBootStage1 panics. It writes a log line through an unmapped address, the same for 4, 8 and 16 GiB of RAM.

Details: bookmark macos-guest-spike, README section "macOS 15, 26, 27".

Assisted-By: Claude Opus 5.5

vivarium: record macOS 15, 26 and 27 on the QEMU fork

vivarium: record headless macOS 15 and 26 on the QEMU fork

Headless works for macOS 15 and 26. nixidae/qemu nixidae 2a4a3122: with gfx=off, QEMU hides the GPU and IOSurface-mapper nodes in the device tree, so no GPU driver loads.

version	first SSH	resets in the 180 s after
15.7.7	48 s	0
26.6.2	25 s	0

The msr S3_6_C15_C8_3 panic is on the GPU path; headless guests never reach it. macOS 27 still fails in iBoot.

Assisted-By: Claude Opus 5.5

Navigation Menu

Repository navigation

vivarium: aarch64-darwin guests; QEMU+HVF cannot answer macOS 13+ PAC hypercalls #1168

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

vivarium: aarch64-darwin guests; QEMU+HVF cannot answer macOS 13+ PAC hypercalls #1168

Description

Activity

Lillecarl commented on Nov 3, 2026

Lillecarl commented on Nov 3, 2026

Lillecarl commented on Nov 6, 2026

Lillecarl commented on Nov 6, 2026

Lillecarl commented on Nov 6, 2026

Lillecarl commented on Nov 6, 2026

Lillecarl commented on Nov 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions