Repository navigation
x86 UEFI boot: an OVMF variable-store reclaim breaks the boot (NVMe times out, "No bootable option") #70
Description
Possibly related
Possible duplicate of #13 and #54; neither has enough information to tell.
- UEFI boot fails with kvm on aws x86 metal instance with #13: black screen, only sometimes, under KVM
- one of the failures below: identical from the outside
- run script: fix intermittent UEFI boot hang on x86_64 #54: "press a key to continue" stall
- fix: keeping
vars.fd - side effect: the store fills up until a reclaim
- fix: keeping
Summary
vars.fd: kept across runs byrun.py, a few entries more with every boot- Full store: a reclaim by the firmware, i.e. a rewrite of all 256 KiB without the deleted entries
- The reclaim boot fails:
- Very slow rewrite
- through QEMU's pflash emulation, one flash write at a time
- boot CPU apparently frozen in
FvbServicesRuntimeDxe: identical RIP and RSP for seconds, no output
- NVMe boot disk timing out afterwards
NvmExpressPassThru: Timeout occurs for an NVMe commandNvmExpressDriverBindingStart: end with Not Found
- No boot entry left
BdsDxe: No bootable option or device was found. Press any key to enter the Boot Manager Menu.- a wait for a key, forever
- with the release OVMF: the only line after the console init, so a silent hang from the outside
- Very slow rewrite
- Transient: a completed reclaim, so a normal next boot from the cleaned store
Hosts
Reclaim and frozen boot CPU on both hosts.
| H | S | |
|---|---|---|
| CPU | AMD Ryzen 7 7700X | AMD Ryzen 7 PRO 5850U |
| OS | Ubuntu 26.04.1 LTS | Fedora 44 |
| Kernel | 7.0.0-38 | 7.2.11 |
| QEMU | 10.2.1 | 11.1.2 |
| OVMF | ovmf 2025.11-3ubuntu7.3, nixpkgs OVMF 202608 debug build |
nixpkgs OVMF 202608 debug build |
On both:
- miniosv master
437621fb4 - an app printing
MINIOSV READY, then power-off - 7 vCPUs, pinned to host CPUs 1–7
Reproducibility
PRO 8000 D: with normal use
- store: 256 KiB
- filled by BDS rewrites of
ConOut,ConIn,ErrOut(plusMTC) on every boot - master
run.pywith-debugconand-monitor: 6–7 entries, ~3.5 KiB per boot, a reclaim every ~70 boots - another setup of ours: ~1 entry per boot, the first reclaim at around 800 boots
REPRO: forced
vars.fdfilled on the host to a few free bytes, with deleted copies of an existing variable- a similar state to the one before a failing boot
- the firmware's next variable write: a reclaim, on every boot from such a store
What we observed
On H:
- Forced reclaim: 6 of 6 failed boots
- both firmware builds
- also with
cache=unsafeon thevars.fdpflash drive, so not the host disk sync
- Store per reclaim: 2,190 deleted entries before, 15 after
- Longest silent stretch: ~20 s (Ubuntu OVMF), ~6–7 s (debug build)
- Register samples during the rewrite (QEMU
info registers, 5 samples): CPU 0 at an identical RIP- Ubuntu OVMF:
FvbServicesRuntimeDxe+0x17BA - debug build:
FvbServicesRuntimeDxe+0x2F16 - both: the instruction right after a byte store to flash (
movb %r13b, (%rbx))
- Ubuntu OVMF:
- Next boots from the reclaimed store: 35 of 35 successful, 0.5–1.7 s each
- Interrupted reclaims: multiple failing boots in a row (
vars.fdnot reclaimed consistently) - Debug log (debug build) of a failing boot:
Ftw: Allocate() success, Caller:FE5CEA76-4F72-49E8-986F-2CD899DFFE5D, # 1
Ftw: Write(), BlockSize - 0x1000, NumberOfWriteBlock - 0x40
Ftw: Write() success, (Lba:Offset)=(0:0x48), Length: 0x3FFB8
[... OMITTED ...]
NvmExpressDriverBindingStart: start
NVMe controller is enabled with status [Success].
NvmExpressPassThru: Timeout occurs for an NVMe command.
NvmExpressDriverBindingStart: end with Not Found
[... OMITTED ...]
BdsDxe: No bootable option or device was found.
BdsDxe: Press any key to enter the Boot Manager Menu.
On S (debug build):
- Register samples during the rewrite: CPU 0 at an identical RIP and RSP,
FvbServicesRuntimeDxe+0x2F16 - CPU 0's QEMU thread during the rewrite: ~78 % CPU, all kernel time
- Other vCPUs: idle (
kvm_vcpu_block) - Forced-reclaim boot:
MINIOSV READYafter ~85 s, no NVMe timeout (1 boot)
Logs, register dumps and the variable stores are available.
Activity
JosVoigt commentedon Nov 1, 2026
Ran a few more boot runs, seems like the root cause of the problem is not a MiniOSv issue but rather the implementation of iouring in QEMU.
There is a chance to ring the NVMe doorbell from mOSv, QEMU registering it but not waking up.
According to my experiments on H this leads to a ~20% failure rate (4 parallel boots though) before keeping the vars.fd (aka pre #54).
Upstream has improvements since 11.x (backported to 10.2.2), but since most of you run Ubuntu 26.04 youre still on 10.2.1.
For me it still failed 3/400 on QEMU 11.1.2 on S, the issue isnt gone.
I was not able to reproduce the issue within 400 runs with the epoll fallback, this seems like the easiest fix.
(Also nearly halved boot time if you care)
QEMU falls back to it if it cannot init the iouring queue, but there is no flag.
This should fix the hang in #70 and probably the root issue of #54 .
Id argue though that we can spare ourselves the 20s of reclamation anyway, so should take a look at vars.fd.
Would be interesting if this is the same issue as #13 as well.
QEMU commits:
https://gitlab.com/qemu-project/qemu/-/commit/961fcc0f22768e7c3432fc645b93dc7cd4932fae
https://gitlab.com/qemu-project/qemu/-/commit/2ae361ef1d7d526b07ff88d854552e2d009bfb1b
Open issue on QEMU:
https://gitlab.com/qemu-project/qemu/-/work_items/4581 (see first comment)
run.py: run QEMU without io_uring to avoid boot hangs (#70)
run.py: use libseccomp for the io_uring filter (#70)
Meandres commentedon Nov 4, 2026
Thanks for opening this issue.
First of all, I think it would benefit from some clarification. I found the bug report rather hard to understand.
If I understand correctly, the problem happens when you boot the same kernel many times with different parameters. While this sounds indeed like a real bug, I would argue that this only happens in a very specific case. I am not sure then that the fix in the run script is warranted.
Maybe you want to revise your benchmarking script to simply rebuild the kernel every 100th boot?
In general, I would say we can keep this issue as a warning for people that might run in the same issue. However, it should be made much more explicit like "Crash when booting many times", with a precise description of (1) what happens, (2) why you think it happens.