You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
<
8000
link rel="modulepreload" href="https://github.githubassets.com/assets/1h3-fe71912fb6b520a1.js" crossorigin="anonymous" />
vivarium: QEMU agent channel dies after ~150 large answers #1178
On a QEMU guest booted without EFI (agent on virtconsole, /dev/hvc0), the agent channel stops answering after a burst of large answers, and never answers again.
Repro, on an idle guest at a pause (solid-kubernetes airlab, vms.airlab-offline.driver -- --offline --break hold):
for n in range(300):
rc, out = await cp.execute("head -c 110000 /dev/urandom | base64 -w 0", timeout=30)
Died after 159 answers (~150 KB each, ~23 MB in 42 s): guest stopped answering during: head -c 110000 .... Every later call fails the same way.
Seen on the guest after the failure (from a host-PID pod, because the agent no longer answers):
vivarium-agent is alive, idle in epoll_wait, no child process.
It is the only holder of /dev/hvc0; the tty is raw (-icanon -isig -ixon -echo -opost).
hvc0 is not a kernel console (console=ttyS0).
So the request is lost between QEMU and the agent, or the reply after it. Found by a phase that polled kubectl get pods -o json once a second; it lost the guest in every run, in the sandbox and by hand.
Activity