Repository navigation
GPU process wedges after ~2.5 days — window stops rendering entirely (PTYs stay alive) #229
Description
What happens
v2.10.1
Summary
After the auto-update from 1.4.0 → 2.10.1, the wmux window stopped rendering completely.
The app was not crashed and not hung at the Win32 level: Responding=True, the window
still accepted and reacted to window messages, and all PTY child processes were alive and healthy.
Only the visual output was frozen — the last painted frame stayed on screen.
Killing the GPU process alone fully restored rendering, with zero loss of panes/PTYs.
This points at the GPU/compositor process rather than the renderer or main process.
Windows' own resource-exhaustion detector (RADAR_PRE_LEAK_64) had already flagged
wmux.exe 2.10.1.0 ~33 hours into the session, before I noticed the freeze.
Environment
| wmux version | 2.10.1 (auto-updated from 1.4.0) |
| Electron | 43.0.0 (per logs/main.log) |
| OS | Windows 11 Pro, Build 22631 (23H2), x64 |
| CPU | 12th Gen Intel Core i7-12800HX |
| RAM | 32 GB |
| Install path | C:\devapps\wmux\wmux.exe (224 MB, mtime 2026-09-08 04:26:02) |
Display adapters (likely relevant — 4 adapters, hybrid + DisplayLink)
| Adapter | Driver | Date | Mode |
|---|---|---|---|
| NVIDIA RTX A2000 8GB Laptop GPU | 32.0.15.9652 | 2026-05-10 | 3440 x 1440 |
| Intel(R) UHD Graphics | 32.0.101.7085 | 2026-03-03 | 2560 x 1600 |
| DisplayLink USB Device | 10.2.7042.0 | 2022-06-18 | 1920 x 1080 |
| DisplayLink USB Device | 10.2.7042.0 | 2022-06-18 | 1920 x 1080 |
The wmux window was maximized across a composite desktop area of 3456 x 1408 at position (-8, -8).
Potentially relevant settings (settings.json)
"wmux-terminal-prefs": { "scrollbackLines": 10000, "fontSize": 16 },
"wmux-appearance-prefs": {
"customBackgroundEnabled": true,
"customBackground": "radial-gradient(...) , radial-gradient(...), #1a1a1a",
"terminalBgOpacity": 30,
"windowTransparency": false,
"windowMaterial": "clear",
"uiMode": "trace",
"hubEnabled": true
}Session size at time of freeze: 28 workspaces, 7 live PTYs, scrollbackLines: 10000.
Timeline (reconstructed from logs + Windows Event Log)
main.log timestamps are UTC; Event Log timestamps are local (UTC+2).
| Time (local) | Event |
|---|---|
| 2026-09-08 04:26:02 | New wmux.exe 2.10.1 written to install dir |
| 2026-09-08 09:50:17 | %LOCALAPPDATA%\wmux-updater\ gets installer.exe + pending |
| 2026-09-08 09:50:26 | main.log: pid=88448 will-quit ptys=48 (1.4.0) |
| 2026-09-08 09:50:28 | Event 1000: wmux.exe 1.4.0.0 crashes, exception 0xc0000409 (fail-fast / stack buffer overrun), faulting module = wmux.exe |
| 2026-09-08 09:50:29 / 09:50:31 | Event 1001, BEX64 buckets for wmux.exe 1.4.0.0 |
| 2026-09-08 09:54:36 | main.log: pid=20112 start version=2.10.1 electron=43.0.0 guard=true — relaunched with --updated |
| 2026-09-09 18:56:41 | Event 1001: RADAR_PRE_LEAK_64, P1: wmux.exe, P2: 2.10.1.0 |
| 2026-09-10 ~18:45 | Rendering observed as completely frozen (~2 days 9 h uptime) |
| 2026-09-10 19:30:52 | GPU process killed → new GPU process spawns, rendering restored |
logs/main.log — note the pattern change
2026-09-05T09:01:42.501Z pid=44360 start version=1.4.0 electron=43.0.0 guard=true
2026-09-06T06:24:39.541Z pid=44360 will-quit ptys=48 guard=true
2026-09-06T16:17:13.097Z pid=86984 start version=1.4.0 electron=43.0.0 guard=true
2026-09-07T06:38:09.542Z pid=86984 will-quit ptys=48 guard=true
2026-09-08T07:49:46.561Z pid=88448 start version=1.4.0 electron=43.0.0 guard=true
2026-09-08T07:50:26.743Z pid=88448 will-quit ptys=48 guard=true
2026-09-08T07:54:36.373Z pid=20112 start version=2.10.1 electron=43.0.0 guard=true
Under 1.4.0 the app was cycled roughly daily and always logged a clean will-quit.
The 2.10.1 instance ran ~2.5 days uninterrupted — the longest single session in the log —
and that is the session that wedged. This may simply be an exposure-time issue
(a slow leak that 1.4.0 users never hit because they restarted more often).
Process state at time of freeze
| PID | Role | CPU (total) | Working Set | Responding |
|---|---|---|---|---|
| 20112 | main | 967 s | 58.2 MB | True |
| 45608 | GPU process | 7,629 s | 35.6 MB | True |
| 39356 | renderer | 4,788 s | 3.2 MB | True |
| 78300 | utility (network) | 11 s | 6.0 MB | True |
| ~30 others | renderers | 2–9 s each | ~3 MB each | True |
Two things stand out:
- The GPU process had accumulated 7,629 CPU-seconds and the main renderer 4,788 CPU-seconds,
while every other renderer sat at 2–9 s. That is 3–4 orders of magnitude more work. - The renderer's working set had been trimmed to 3.2 MB — consistent with a process that has
not painted for a very long time and got paged out.
Crucially, at the moment of investigation all of these were at 0% CPU. This is not a busy loop —
it is a wedged/stalled compositor. Measuring CPU deltas over a 6 s window showed 0.0% for
GPU, renderer and main alike.
Diagnostics performed (ruling out the usual suspects)
| Hypothesis | Measurement | Result |
|---|---|---|
| Runaway CPU loop | CPU delta over 6 s, all processes | 0% across the board — ruled out |
| Output flooding a pane | I/O byte delta over 5 s, all shells | 0 MB/s — ruled out |
| GDI/USER handle exhaustion | GetGuiResources() per process |
5,484 GDI / 5,930 USER system-wide (limit 10,000 per process) — ruled out |
| GPU driver reset (TDR) | Event Log ID 4101/4104 | no events — ruled out |
| Hung filesystem/mount blocking paint | rclone/WinFsp/Dokan process+service scan | none present — ruled out |
| Window off-screen / minimized | GetWindowRect, IsIconic |
rect (-8,-8)–(3448,1400), not iconic — ruled out |
| Window-level hang | Responding, Event 1002 (App Hang) |
True, no hang events — ruled out |
Forced repaint did not help
Sending RedrawWindow(RDW_INVALIDATE|RDW_ERASE|RDW_ALLCHILDREN|RDW_UPDATENOW|RDW_FRAME)
followed by a ShowWindow(SW_MINIMIZE) → ShowWindow(SW_RESTORE) cycle produced no repaint
and no measurable CPU in GPU/renderer/main.
Interestingly the working sets did grow in response (GPU 30.0 → 35.6 MB, main 52.9 → 58.2 MB),
so the window was receiving and processing the messages — it allocated, but never painted.
This strongly suggests the compositor accepted the invalidation but could not produce a frame.
Workaround / fix that worked
Killing only the GPU process:
Stop-Process -Id <gpu-process-pid> -ForceResult — immediate and complete recovery:
New GPU PID 59988, started 19:30:52, WS = 116.2 MB, CPU = 0.6 s
- Rendering came back instantly.
- All 7 pane shells survived. All 6 nested agent processes survived.
- No data loss, no PTY loss.
Note the contrast: the fresh GPU process sits at 116 MB working set, whereas the wedged
one had been trimmed to 35 MB. The old one wasn't rendering, it was spinning its wheels.
This is a much better remedy than a full restart, because wmux reaps every PTY listed in
pty-ledger.json on quit ({"ownerPid":20112,"entries":[...8 pwsh entries...]}), which would
have destroyed all running sessions.
Suggested investigation areas
- GPU process resource leak in 2.10.1. Windows'
RADAR_PRE_LEAK_64fired onwmux.exe 2.10.1.0
specifically. Combined with 7,629 CPU-seconds on the GPU process, a leaked GPU
resource/command-buffer per frame or per PTY write seems plausible. - Compositor interaction with
terminalBgOpacity/customBackground/windowMaterial: "clear".
Per-frame blending of two radial gradients behind 30%-opacity terminal content across a
3456x1408 surface is a heavy compositing path. Worth testing with these disabled. - DisplayLink + hybrid GPU. Two DisplayLink adapters on a 2022 driver alongside NVIDIA/Intel
switchable graphics is a known-difficult configuration for Chromium's compositor. Surface loss
on a DisplayLink adapter that is never recovered would match the symptoms exactly. - A watchdog would help. If the GPU process produces no frame for N seconds while
invalidations are pending, wmux could self-heal by restarting its own GPU process — exactly the
manual action that fixed this. Given PTYs live in the main process, this is safe to do
automatically and would be invisible to the user. - Scale factor. 28 workspaces ×
scrollbackLines: 10000may be relevant to leak rate.
Suggested feature request (separate but related)
Make the auto-update opt-in, or at least deferrable.
The update applied itself unattended and jumped 1.4.0 → 2.10.1, a major version change.
The old binary died with 0xc0000409 during the process. For a terminal multiplexer holding
long-lived sessions this is high-stakes: had I restarted the app to fix the freeze, I would have
lost 7 live shells with unsaved work — the pty-ledger.json reaping is unconditional on quit.
Two things that would help a lot:
- A "defer update until I quit" option (never swap the binary under a running instance).
- Session/PTY re-attach across restarts, so a restart is not destructive.
pty-ledger.json
already tracks the PIDs; adopting rather than reaping them would make wmux dramatically safer.
btw
25 minutes after killing only the GPU process, the original renderer process 39356 was still
running — never restarted — and had gone from a trimmed 3.2 MB back up to 303.9 MB working set:
timestamp,role,pid,cpu_total_s,cpu_pct_sample,ws_mb,handles,threads,uptime_h
2026-09-10T19:55:03,gpu,59988,23.84,,62.7,678,60,0.4
2026-09-10T19:55:03,renderer,39356,4811.55,,303.9,499,37,58.01
2026-09-10T19:55:03,main,20112,970.47,,71.5,2711,119,58.01
This is strong evidence that the renderer was never the broken component. It was healthy the
whole time and simply starved, unable to get frames presented by a wedged GPU process. The moment
a functioning GPU process existed, it resumed normally, repopulated its working set, and kept its
existing PID and state. The fault is isolated to the GPU/compositor process.
Steps to reproduce
there are no deterministic steps i know of to reliably reproduce this
wmux version
v2.10.1
Windows version
Windows 11 Pro, Build 22631 (23H2), x64
Anything else
No response
Activity
fix(gpu): self-heal a wedged GPU process; update dialog defaults to L…
amirlehmam commentedon Sep 11, 2026
Thank you for this report. It is one of the best-instrumented issues wmux has had: the per-process CPU deltas, the working-set trim on the renderer, the forced-repaint experiment and the "kill only the GPU process" recovery together pin the fault to the display compositor in Chromium's GPU process, and the "btw" section (same renderer PID, 3.2 MB → 304 MB the moment a GPU process existed) closes the case on the renderer being healthy the whole time.
I could not reproduce it, and I do not think anyone will on demand: a hybrid NVIDIA/Intel laptop with two DisplayLink adapters on a 2022 driver, a 3456×1408
clearwindow, and 58 hours of uptime is a combination I cannot build a test around, and the wedge is inside Chromium rather than in anything wmux draws. So 2.10.4 ships your suggestion #4 rather than a root-cause fix: wmux now notices the condition and applies exactly the remedy you applied by hand.What 2.10.4 does
Detection.
requestAnimationFrameis driven by BeginFrames from the display compositor in the GPU process, so when that process stops producing frames, rAF stops firing whilesetTimeoutkeeps running on the (healthy) renderer main thread. The renderer requests a frame every 5 s and checks whether the previous one arrived. Three unanswered requests while the document is visible is a report to main. Your process data says the same thing from the other side: a renderer with agents streaming into seven panes had 0% CPU and a trimmed working set, i.e. its frame loop was not running at all.Decision, in main. The renderer never acts on its own. Main restarts the GPU process only when all of these hold:
GetLastInputInfovia Electron'spowerMonitor), so a locked or unattended machine can never trip it,In practice this means the heal happens within about 15 s of you clicking into a frozen window, which is precisely the moment you would otherwise reach for
Stop-Process.The restart is
process.killon theGPUentry ofapp.getAppMetrics(), the same thing you did. Chromium relaunches the process on demand; every visible terminal loses its WebGL context and goes through the context-loss path from #218 (drops to the DOM renderer, repaints, picks WebGL back up on its next show). PTYs live in the main process and are untouched. A bell notification tells you it happened.Diagnostics.
%APPDATA%\wmux\logs\main.lognow recordsgpu-stall(with the reason a report was skipped, deduplicated per run),gpu-restart, andgpu-gonefor every GPU process exit with Chromium's reason code. If it ever fires on your machine, those lines pluswmux crash-reportare what I would like to see. Off switch if you need it:"gpuWatchdog": falseunderwmux-workspace-prefsinsettings.json, read live.The update prompt
wmux does not install updates silently:
autoDownloadandautoInstallOnAppQuitare off and the installer only runs after a native "Install and restart / Later" dialog. But that dialog's default button was Install and restart, and it opens over a terminal you are typing into, where Enter is the most common key. I suspect that is how 1.4.0 → 2.10.1 "applied itself". As of 2.10.4 the default is Later; installing takes a deliberate click. The0xc0000409on the way out is #214's shutdown race, addressed in 2.8.0; 1.4.0 predates that fix.Not in this release
pty-ledger.jsonis half of what it needs. It is a real feature rather than a fix, so I have not bundled it here; it deserves its own issue if you want to open one.RADAR_PRE_LEAK_64nameswmux.exe, but every Chromium child is alsowmux.exe, so it does not say which process. If you are willing to keep theclearmaterial + 30% opacity + gradient background on, thegpu-gonelines will tell us whether Chromium's own watchdog ever kills the process before wmux does. If you are willing to turn them off for a few days, that is the A/B on your suggestion feat: improve terminal and markdown link behavior #2 that I cannot run here.2.10.4: https://github.com/amirlehmam/wmux/releases/tag/v2.10.4