Renderer private bytes reach 6.2 GB after 92 h, and are still climbing on 2.51.0 (follow-up to #385)

Follow-up to #385, which was closed as fixed in 2.46.0. I kept measuring
afterwards. The retention is still there, and I now have a far better dataset
than the one I gave you last time. I can't reopen a closed issue as an outside
contributor, so this is a new one.

Short version: on 2.47.0 the renderer went from 253 MB at startup to
6183 MB of private bytes after 92 hours, with the final reading taken while
the machine was idle. On 2.51.0 it is better behaved but still climbing:
262 MB → 1160 MB over 25 hours.

2.47.0, sampled by hand

private is PrivateMemorySize64. load is the instantaneous CPU across all
node processes at the moment of sampling.

hours	RSS (MB)	private (MB)	load
0.02	253	-	idle
19.97	344	-	idle
22.61	566	-	7 %
24.35	502	-	idle
44.60	1259	1336	28 %
91.98	5467	6183	idle

Roughly 24 MB/h over the first 44 hours, then roughly 102 MB/h from hour 44 to
hour 92. The growth accelerates rather than flattening, which is the opposite of
what a cache filling up looks like. The last row is the important one: the
machine was idle, nothing was in flight, and the renderer still held 6.2 GB.
Only restarting the app brought it back to baseline.

2.51.0, sampled hourly

Same machine, same kind of workload, a scheduled task taking one sample an hour.
26 samples so far.

hours	RSS (MB)	private (MB)	load
0.02	317	262	idle
1.26	676	672	5 %
2.29	517	540	3 %
4.29	532	620	idle
6.29	783	844	3 %
8.29	831	875	idle
12.29	945	981	idle
15.29	1044	1078	idle
18.29	1015	1047	idle
20.29	1018	1050	idle
22.29	1018	1050	idle
24.29	1097	1159	idle
25.29	1097	1160	idle

Two things changed for the better, and one did not.

It releases now. 672 → 540 MB in the second hour, with smaller drops later.
My 2.47.0 log never does that; it only goes up. This is consistent with the
release fix you shipped in 2.46.0 doing something real.

It plateaus. Hours 15 through 22 sit inside a 31 MB band: 1078, 1075, 1058,
1047, 1048, 1050, 1048, 1050. That is genuinely flat.

But it is higher than 2.47.0 was at the same age. At 22 hours: 1018 MB RSS
on 2.51.0 against 566 MB on 2.47.0 at 22.6 hours. The workload was not identical
(25 node processes now against 19 then), but that does not account for a
doubling. And after the plateau it starts climbing again: 1050 → 1090 → 1159.

Why I am filing this now rather than waiting

Because 25 hours cannot answer the question that matters, and I want to be
straight about that rather than overstate what I have.

On 2.47.0 the 24-hour mark looked fine too: 502 MB. The collapse happened
between hour 44 and hour 92, where it went from 1336 MB to 6183 MB. So a day of
data understated the end state by a factor of twelve. The 2.51.0 plateau may be
your fix working. It may also be exactly what 2.47.0 looked like before it took
off. I cannot tell yet.

The hourly sampling is still running. I will come back and post the 48-hour
and 92-hour readings for 2.51.0, which is where the old curve broke down. If
it stays flat, I will say so plainly here and you can close this.

A correction to what I told you on #385

In my comment on #385 I reported a 635 → 504 MB drop as evidence that your fix
released memory. I cannot stand behind that reading. I had no private-bytes
number next to it, and under memory pressure Windows trims the working set, so
RSS can fall while nothing has actually been released. One sample in this run
shows that clearly: at 16:14, under 40 % load, RSS read 136 MB while private
bytes were 597 MB. Logging RSS alone would have recorded that as the renderer
releasing 500 MB. It had released nothing.

Every number in both tables above has private bytes next to it for that reason.

Ruled out

The notification array. At the 24-hour sample on 2.47.0 it held 69 entries,
0 unread, arriving at about 3.1/hour. That cannot account for hundreds of MB, so
the notification-retention fix is not where the residual lives.

What I could not do

No heap snapshot. The CDP endpoint on port 9222 listens and answers
/json/version and /json/list, and the WebSocket connects, but no CDP method
ever responds: Runtime.evaluate, HeapProfiler.enable and Target.getTargets
all time out after 6 s with no response and no error frame. That matches #157.
Point me at a build where the CDP proxy actually binds and I will take a heap
snapshot at the 48-hour mark and attach it.

Environment

wmux 2.47.0 and 2.51.0, Windows 11 Pro 22000, 32 GB RAM
One window, two to three workspaces, an agent CLI running in each pane
Sampler reads the --type=renderer process and logs RSS, private bytes, free
physical memory, commit charge and node CPU load in the same row

Thank you for the detailed measurements and for correcting the earlier RSS interpretation. The 2.51.0 private-byte series is useful, but it does not yet establish whether the long-run growth is fixed, and I will keep this issue open while the 48-hour and 92-hour measurements come in.

The endpoint on port 9222 is wmux's browser-panel CDP proxy, not the main wmux renderer, so it cannot produce a heap snapshot of the terminal UI. You can instead open the main window's developer tools from Settings > Help > Toggle Developer Tools, then use the Memory tab to take heap snapshots at two comparable idle points. A snapshot can contain terminal output, prompts, paths, or other sensitive data; please inspect it and share it only if you are comfortable doing so. Aggregate constructor/retainer differences without the raw snapshot are also useful. Please record the same renderer PID, private bytes, works 10000 pace/terminal count, and workload alongside each snapshot.

A heap snapshot measures the JavaScript heap, not all Chromium native or GPU memory. If the JS heap stays flat while private bytes continue to rise, that would point the investigation elsewhere. I have not identified the residual retention path yet and am not calling this fixed.

54-hour update, and it corrects the picture I was about to give you.

I had a 48 h comment written this morning saying 2.51.0 had gone flat. I did not post it, because the flatness turned out to be an artifact of an unattended machine. Work resumed here about seven hours ago and the growth came back.

Renderer PID 3556, one process throughout, no restart. One window, two to three workspaces, an agent CLI in each pane.

hours	private (MB)	change	node CPU at sample	node procs
30.29–47.29	1220–1223	flat	0 %	25
48.29	1714	+492	0 %	25
49.29	1365	−349	0 %	25
50.29	1362	−3	0 %	25
51.29	1718	+356	26 %	42
52.29	1404	−314	1 %	27
53.29	1503	+99	0 %	25
54.20	2000	+497	0 %	25

One caution about that CPU column, since you asked for workload alongside each reading: it is a three-second instantaneous sample taken at the moment of the reading, so it says nothing about the hour it sits in. Hours 30 to 47 were genuinely idle — the machine was unattended, which I can vouch for independently of the counter. The zeros from hour 48 onward are just quiet instants inside a working day and should not be read as idle hours.

Two things the series says.

It still releases. 1714 → 1365 and 1718 → 1404: roughly 320 to 350 MB handed back after a burst. My entire 2.47.0 log never once decreased. That part of the 2.46.0 fix is doing real work.

But the floor steps up rather than returning. The post-release floors read 1365, 1362, 1404, and the overall move is 1222 MB at 47.29 h to 2000 MB at 54.20 h — 778 MB in 6.9 hours, about 113 MB/h. For comparison, 2.47.0 went from 1336 MB at 44.6 h to 6183 MB at 92 h, about 102 MB/h. So the rate while the machine is actually working is not better than the old one. What is better is that it gives some back between bursts, which may mean it plateaus somewhere the old one never did. Seven hours cannot tell me where.

Treat the 2000 MB row as a peak rather than a settled floor: it is the newest reading and has not had an hour to release yet. On the pattern above it will probably come back to somewhere near 1600 to 1700.

Where that leaves it at 54 hours: the unbounded growth of 2.47.0 looks damped, not gone. 2000 MB against a 262 MB startup baseline is 7.6x after a little over two days, and the daytime rate is in the same range as the version you fixed.

I will still post the 92 h reading on Wednesday morning, and that one will have a full working day behind it rather than a quiet night.

Two notes on your reply:

Thank you for the correction on port 9222. I had it filed as a dead proxy matching #157, and that it belongs to the browser panel rather than the terminal renderer explains cleanly why no CDP method ever answered. Connecting it to that issue was my mistake.

On the heap snapshots: I can reach the machine again now, so I will take them from Settings → Help → Toggle Developer Tools → Memory as you described. Understood about the contents — I will post aggregate constructor and retainer differences rather than the raw snapshot, and record renderer PID, private bytes, workspace and surface counts and workload alongside each. Given the shape above I will aim for one at a post-release floor and one at a peak, which should separate what is held from what is in flight better than two readings at the same point would.

Thanks for the 54-hour series. 2.55.0 has no change for this, on purpose. I went through every piece of renderer state that grows with output or hook events: prompt log, markers and highlights, screen events, titles, subagents, notifications, touched files, per-surface maps. All of it is capped or dropped when a surface closes, and I found nothing that grows without bound. I don't want to ship a speculative change.

The best remaining suspect is native, not JS: xterm's WebGL glyph atlas. It is shared by every visible terminal with the same font and theme, and it keeps one entry per character/foreground/background combination. Its pages merge into larger canvases outside the JS heap. Nothing clears it while a visible terminal still uses it, and with agent panes on screen all day that never happens. A merge allocates the bigger page before it frees the small ones. That would give a "+N then −M" curve like your +492/−349 and +356/−314 steps. I can't yet show it is big enough to reach several GB, though.

What would separate the two at your next readings, at a floor and at a peak:

JS heap next to private bytes. In the main window's DevTools console, run performance.memory.usedJSHeapSize, or read the Memory tab's heap size. If the JS heap stays flat while private bytes climb, wmux's own state is ruled out.
Aggregate snapshot diff. Compare constructor counts and sizes, especially HTMLCanvasElement, CanvasRenderingContext2D, ImageData and ArrayBuffer. Growth in canvases points at the glyph atlas.
How many panes are visible at once. Only visible panes get WebGL, so this tells me how many terminals keep the shared atlas alive.

If (1) shows a flat JS heap, I'll add a way to run terminals on the DOM renderer, so you can compare a working day's rate directly.

JS heap is ruled out, and the console named the mechanism itself.

Renderer PID 3556, 74.8 h uptime, no restart, machine under normal agent load throughout (42 to 44 node processes).

1. JS heap next to private bytes

15:47   private 1365 MB
15:50   private  559 MB     performance.memory.usedJSHeapSize = 47400000
15:53   private 1120 MB     performance.memory.usedJSHeapSize = 47400000

I cannot pin each heap read to one specific private-bytes row — both were taken inside that same six-minute window — but that is enough. The JS heap read identical to the byte at both ends while private bytes fell 806 MB and came back 561 MB on their own. 45.2 MB of JS heap against native memory that swings by 800 MB in minutes.

Caveat where it is due: performance.memory is bucketed, so two identical readings are not surprising at this granularity. The magnitude is the argument, not the equality. 45 MB cannot account for 800 MB of movement, and it cannot account for the 262 MB → 2020 MB climb either. Your own JS state is not where this lives.

2. The console says it outright

First line in the renderer console, with 17 occurrences:

WARNING: Too many active WebGL contexts. Oldest context will be lost.

3. Visible panes

One window. Four workspaces, one visible at a time. The visible workspace holds 8 surfaces, and several of those are markdown views rather than terminals, so visible terminals are well under ten and have never been near sixteen.

Those two facts together are the finding. Fewer than ten visible terminals, and Chromium is still hitting its per-renderer cap on active WebGL contexts, repeatedly. The contexts are outliving visibility — kept across workspace switches, or not released when a surface closes. Your note said only visible panes get WebGL; on this machine that does not appear to hold.

It also fits the shape of the curve. A context that gets force-lost at the cap and then recreated has to rebuild its glyph atlas from nothing, and the rebuild allocates before the old pages go. That is the "+N then −M" you predicted, and it would repeat every time the cap is crossed rather than only when fonts or themes change.

And it fits the largest single event in my log, which I have not reported before. Earlier today, same process, no restart:

11:14   2020 MB
12:14    899 MB
13:14    691 MB     1329 MB released over two hours

My entire 2.47.0 log never dropped once. A mass release on that scale, with a 45 MB JS heap, is not garbage collection — it is native allocations being dropped, which is what should happen when contexts are finally let go.

Still to come

The heap snapshot. I will post aggregate counts for HTMLCanvasElement, WebGLRenderingContext, CanvasRenderingContext2D, ImageData and ArrayBuffer rather than the raw file. If the retained canvas count is far above the number of terminals actually on screen, that closes it.

If it would help, tell me what you want logged around a workspace switch specifically — that is the cheapest way to catch a context being created without one being destroyed, and I can run it on demand.

Heap snapshot taken at a peak. The canvases account for it, and the JS heap never could have.

Conditions at the moment of the snapshot: renderer PID 3556, 74.94 h uptime, no restart, private bytes 2077 MB, RSS 1048 MB (Windows had trimmed the working set; private is the real figure). 16 % node CPU across 29 processes. One window, four workspaces with one visible, 8 surfaces in the visible one of which several are markdown views rather than terminals.

Snapshot: 1 324 632 nodes, 3 878 486 edges, 82.6 MB total self_size.

Canvas backing store

Every <canvas> in the snapshot, with its dimensions read off the element and multiplied out at RGBA8:

count	size	backing store
24	1407×1290	166.2 MB
19	1589×1290	148.6 MB
10	1127×1290	55.5 MB
6	1596×1290	47.1 MB
34	512×512	34.0 MB
9	686×1290	30.4 MB
3	1855×1290	27.4 MB
4	10×1290	0.2 MB
34	32×23	0.1 MB
2	1×1	0.0 MB

145 canvas elements, 509.4 MB of backing store, for one CPU copy. A composited accelerated canvas generally also has a GPU-side copy, so the true cost is plausibly around a gigabyte against the 2077 MB measured. None of this appears in usedJSHeapSize, which is why it read 45 MB while private bytes swung by 800 MB, and why auditing JS-side state found nothing.

The 34 canvases at 512×512 are your glyph atlases: exactly one per WebGL context, 34 MB in total. Your instinct was right about where to look, but the atlas is the small part. The bulk is 71 full-size terminal render surfaces.

Object counts

 34  WebGL2RenderingContext
111  CanvasRenderingContext2D
 16  WebGLTexture
  4  OffscreenCanvas

34 live WebGL2 contexts, against a Chromium cap of 16 and the 17 "Too many active WebGL contexts" warnings from the previous comment. Against 8 surfaces in the one visible workspace. Four workspaces exist in total, so 34 is close to what you would get from one context per pane across all workspaces rather than per visible pane.

That also shows up in the widths: 1407, 1589, 1127, 1596, 686 and 1855 pixels are six different pane layouts, all holding full-height canvases at the same moment. Some of that is legitimately four workspaces with different splits. 24 canvases at a single width is harder to explain that way, since two canvases per terminal puts twelve terminals at that one width.

Detached DOM

1729 detached nodes, including four complete retained terminal subtrees:

4  Detached kept-terminal-slot
4  Detached terminal-pane__container
4  Detached xterm-viewport
4  Detached xterm-screen
7  detached canvas elements, 11.3 MB of backing store

Four whole terminal instances removed from the document and still held. Small next to the 509 MB, but it is an unambiguous retention path and probably the easiest of the three to chase down.

What I would take from this

Three separate things, in order of how much memory they hold:

Full-size terminal canvases are retained per pane across all workspaces, not per visible pane, and across layout changes. This is the 500 MB.
WebGL contexts track the same population: 34 of them, over twice the browser's cap, with contexts being force-lost and rebuilt as a result.
Four terminal DOM subtrees leaked outright.

The kept-terminal class name suggests (1) is deliberate, to keep terminals warm across workspace switches. If so, the bug is not that they are kept but that nothing bounds the total or drops the backing store for panes that have not been on screen in hours.

Happy to run a floor snapshot tomorrow morning after the overnight flat stretch and diff the two, which should separate what is genuinely permanent from what is in flight. Tell me if you want anything else counted in the meantime — I have the file locally and can query it without bothering you for another capture. Not posting the file itself, for the reasons you gave.

Quick controlled test while I had the machine in this state, since it settles the open question from my last comment.

I closed one workspace that had been sitting idle, sampling the renderer every six seconds. Same process, PID 3556, nothing else changed.

16:00:39   private 1260 MB
16:00:45   private  999 MB    <- workspace closed
16:00:51   private  999 MB
16:01:03   private 1001 MB
16:01:27   private 1000 MB

Flat for 18 s before, one step of −261 MB, flat for the next 48 s, then a slow creep back up as work continued in the remaining workspaces.

So the retention really is per workspace, and it is held for workspaces that are not visible and are doing nothing. 261 MB for one idle workspace, which lines up with the canvas accounting in the previous comment once GPU-side copies are included.

Two workspaces here are still idle, so by this measure roughly half a gigabyte of the current total is being held for panes nobody is looking at.

It is also a usable workaround in the meantime, which is worth something to anyone hitting this: closing an idle workspace gives the memory back immediately.

Correction to my previous comment: the 261 MB did not replicate, so withdraw the extrapolation.

I closed a second idle workspace, same method, same process, sampling every six seconds:

16:07:33   private 1313 MB
16:08:27   private 1313 MB
16:08:33   private 1314 MB    <- workspace closed; RSS -70 MB, all wmux processes -105 MB
16:10:25   private 1313 MB
16:11:07   private 1317 MB

Private bytes did not move. RSS fell 70 MB and the sum across all wmux processes fell 105 MB, and roughly half of that came back within two minutes.

So the first result stands as an observation but not as a rate. One workspace gave back 261 MB in a clean step; the next gave back essentially nothing. "Roughly 260 MB per idle workspace", and the half-gigabyte estimate I derived from it for the remaining workspaces, are not supported. Please disregard both.

I do not know yet what differs between the two. The obvious candidate is pane count — the first may simply have held more terminals than the second — but I did not record it before closing them, which I should have. If it matters to you I can set up the next one properly: record pane count and canvas inventory first, then close, then re-snapshot.

What this does not touch is the snapshot accounting in the comment above. 145 canvas elements and 509 MB of backing store were read directly out of the heap snapshot, with the dimensions on the elements themselves, at a moment when private bytes were 2077 MB. That number does not depend on the close experiment and still stands, as do the 34 WebGL2 contexts against a cap of 16, and the four detached terminal subtrees.

Sorry for the noise. I would rather retract it the same hour than leave you working from it.

Follow-up on my own correction, with the number I was missing.

I asked how many panes each of the two closed workspaces held. The first had 4 to 5; the second had 1. That accounts for the difference, and it puts the result back together as a per-pane effect rather than a per-workspace one.

4-5 panes  ->  private -261 MB      ~55-65 MB per pane
1 pane     ->  private unchanged, RSS -70 MB

The single-pane close did free memory — RSS fell 70 MB, which is right in line with that per-pane figure. What it did not do is hand the commit back. That is ordinary allocator behaviour: a large free gets returned to the OS, a small one stays committed for reuse. Which also means private bytes understate release for small frees, the mirror image of the working-set trap I have been warning about in the other direction all along.

So, restated carefully: closing panes frees roughly 55 to 65 MB each, and that is consistent with the canvas accounting. A terminal at 1407×1290 carries two canvases at 6.9 MB each, plus a GPU-side copy, plus the context overhead and its 512×512 atlas — call it 30 to 40 MB of directly attributable surface, with the rest presumably in command and transfer buffers.

Two caveats I want on the record. The pane counts are from memory after the fact, not recorded before closing. And n=2. I am not offering 55-65 MB as a measured constant, only as the figure these two trials are consistent with.

My earlier "260 MB per idle workspace" remains withdrawn. Per workspace was the wrong unit.

fix(terminal): release a hidden terminal's WebGL context immediately (#…

Thank you. The snapshot accounting found it: 34 WebGL2 contexts with fewer than ten terminals on screen, next to a 45 MB JS heap.

wmux already released a terminal's WebGL renderer when the terminal was hidden and capped live renderers at 12, so the cap was never the problem. The problem is what WebglAddon.dispose() leaves behind: in addon-webgl 0.19.0 it removes its canvas from the page and nothing more. The WebGL2 context and the full-size drawing buffer stay alive until the canvas is garbage collected. wmux disposes that renderer on every workspace switch, every parked pane and every settled width change, so dead contexts pile up between collections. Chromium counts them against its 16-context limit, and that produces your "Too many active WebGL contexts" warnings.

I reproduced it with a real xterm in a browser: 30 hide/show cycles on one terminal. Before the fix, 15 of the dead contexts were still alive and Chromium had force-lost the other 15. 2.55.1 now calls WEBGL_lose_context and resizes the canvas to 0×0 right after dispose. With that, all 30 are released immediately, and the visible terminal stays on WebGL with nothing force-lost. This works even if something still holds a reference to the old canvas, so it doesn't depend on finding every retainer.

What this doesn't cover yet:

The four detached terminal subtrees (kept-terminal-slot, xterm-screen, …). I haven't found 10BC8 what holds them, so they aren't fixed. If you can query your snapshot for the retainer path of one detached kept-terminal-slot (the first few non-internal retainers are enough), that would show where it's held.
The 71 full-size canvases. The fix shrinks the canvases of discarded WebGL renderers, which should account for most of them, but I'm leaving the issue open until your numbers show it.

On 2.55.1, a peak/floor pair like your last one would settle it: private bytes, performance.memory.usedJSHeapSize, and the counts of WebGL2RenderingContext and full-size HTMLCanvasElement in a snapshot. Live contexts should now stay at about one per visible terminal, and the warning should no longer appear.

Retainer paths for the detached kept-terminal-slot nodes, from the peak snapshot. One important caveat first, because it affects what you should trust in my earlier numbers.

The snapshot was taken with DevTools open, and DevTools retains DOM nodes

For two of the four, the shortest path to a GC root runs through DevTools' own node map:

C++ Persistent roots
  -> blink::DevToolsSession
    -> blink::InspectorDOMAgent
      -> HeapHashTableBacking<HashTable<int, KeyValuePair<int, ...>>>
        -> Detached <div class="kept-terminal-slot">

That is the inspector holding every node it has seen, and it exists only because I had DevTools open to take the snapshot. So the detached-node count I reported (1729, and the four subtrees) is inflated by my own instrumentation and should not be treated as a clean leak count. I should have flagged that possibility when I posted it.

The other two reach a root through the frame view's scrollable-area registry instead:

blink::WebLocalFrameImpl -> LocalFrame -> LocalFrameView
  -> HashTable<cc::ElementId, ...> -> PaintLayerScrollableArea -> PaintLayer
    -> LayoutBlockFlow -> Detached <div class="terminal-pane ">
      -> Detached <div class="kept-terminal"> -> Detached <div class="kept-terminal-slot">

The JS-side retention, which is the same for all four and is not a DevTools artifact

Ignoring Blink internals, every one of the four is held by exactly two kinds of JS reference:

Detached <div class="kept-terminal-slot">
  <- property:stateNode   L [object]          React fiber
       <- property:child      L
       <- property:alternate  L               the alternate fiber tree
  <- property:current     Object [object]     a ref object
       <- property:ref         Object
       <- property:memoizedState Object       useRef hook state

Two mechanisms, both React-side:

A ref still points at the removed node. {current: <div class="kept-terminal-slot">}, reached from memoizedState on a hook and from a ref property. Nothing nulls .current when the kept terminal is torn down, so the ref pins the whole subtree.
Alternate fibers. The fiber holding stateNode is reachable via alternate, so React's previous tree keeps the old DOM pointer alive past the unmount.

(2) normally resolves itself on a later render. (1) does not: a ref whose .current is never cleared keeps the node for as long as the component holding the hook lives. If the hook lives on something longer-lived than the terminal — a workspace or pane wrapper that survives the terminal being parked — that is a permanent hold, and four of them matching four parked terminals fits that shape.

Cheapest thing to try: null the ref in the teardown path for a kept terminal.

On 2.55.1

I will update and post the peak/floor pair you asked for: private bytes, usedJSHeapSize, WebGL2RenderingContext count and full-size canvas count at both ends. I will take the floor snapshot with DevTools opened only at the moment of capture, and I will say so, so we can compare the detached counts on a like-for-like basis rather than against today's inflated one.

fix(renderer): stop the WebGL cursor-blink interval when releasing th…

test: unstub requestAnimationFrame in afterEach (amirlehmam#421)

fix(renderer): stop the WebGL cursor-blink interval when releasing th…

@ConnorRobinson13

2.55.2 is out with the fix from #433 (thanks to @ConnorRobinson13), and I think it is the climb you have been measuring.

In addon-webgl 0.19.0 the WebGL renderer creates a cursor-blink manager and never disposes it (xtermjs/xterm.js#5818, fixed upstream but not in a stable release). Its 600 ms setInterval survives dispose(). Through that timer the renderer stays reachable, and through the renderer the whole Terminal, scrollback included. wmux releases the renderer on every hide, park and close, so each of those leaked one terminal if it had focus. That fits everything you reported: memory grows while you work and stays flat when wmux sits idle. It is mostly native memory. And closing workspaces gives little back, because the leaked terminals were already detached from them.

I confirmed the cause against the shipped 0.19.0 build: the manager is created with new and never _registered. Connor's one-hour soak (3×3 panes, one workspace closed and reopened per cycle) went from about +14 MB post-GC private memory per cycle to flat, and from 15 xterm cores alive with 9 terminals open to 9.

On the detached kept-terminal-slot paths: thanks for separating the DevTools-held ones out. For the other two, I don't think a ref that's never nulled is the holder. React 19.2 sets ref.current = null on every deleted host node, unless the node is inside a hidden <Activity> or Suspense subtree, and wmux has neither. The keep-alive also moves the terminal out of a slot before React removes that slot. A {current: slot} that still points at a detached slot with a terminal inside it therefore looks like fibers React has not collected yet, or a mid-commit capture. It doesn't look like a permanent hold. Four small DOM subtrees also can't account for the gigabytes, while leaked terminals can. So I haven't changed anything there. If the count grows with uptime on 2.55.2, that would change my mind.

The peak/floor pair you offered on 2.55.2 would settle it: private bytes, usedJSHeapSize, and WebGL2RenderingContext and full-size HTMLCanvasElement counts. If you can, add the number of xterm Terminal/CoreTerminal instances in the snapshot next to the number of terminals open. That's the number this fix should bring down. I'm keeping the issue open until your numbers come in.

92-hour reading on 2.51.0, as promised. I have not updated yet, so this is the clean "before" baseline for the 2.55.2 numbers.

version	hours	RSS (MB)	private (MB)
2.47.0	91.98	5467	6183
2.51.0	92.29	1731	2241

2241 MB against 6183 MB at the same age. 2.51.0 still never returns to its 262 MB start and is still climbing (2267 MB at 97.3 h), but the acceleration that took 2.47.0 from 1336 MB to 6183 MB between hour 44 and hour 92 is gone.

The load/idle split matches your description

window	hours	node CPU (sampled)	private bytes
18:14-23:15	5	24-36 %	1171 -> 1787 MB (+123 MB/h)
00:14-05:14	5	0-7 %	1796 -> 2238 MB (+88 MB/h)
05:14-12:14	7	0 %	2238 -> 2244 MB (+0.9 MB/h)

The last row is the flattest stretch in the entire run: seven hours inside 6 MB. That is your "flat when wmux sits idle", and the first row is "grows while you work".

The middle row does not fit, and I would rather flag it than smooth it over. 442 MB appeared between 00:14 and 05:14 with node CPU sampling at zero. That column is one instantaneous sample an hour, so it misses work easily - but 442 MB is a lot to put on sampling error. Either something ran that I did not see, or something grows without foreground work. Hourly samples cannot separate those two.

Renderer count from the peak snapshot

I re-counted the 15:55 snapshot for the numbers you asked for:

what	count
`WebGL2RenderingContext` (native)	34
512x512 glyph atlas canvases	34
full-size terminal canvases (height 1290)	71
all canvases	145, 509.4 MB of backing store (one CPU copy)

Terminals open at the time: 14 or 15, across four workspaces.

I cannot give you the xterm Terminal/CoreTerminal count. The shipped build is minified, so those constructor names are not in the snapshot at all - my scan returns zero, which means absent, not collected. The glyph atlases are the usable proxy instead: 34 atlases to 34 contexts is exactly 1:1, so 34 renderers were alive against 14-15 terminals. Same shape as Connor's 15 cores to 9 terminals, somewhat worse.

If you ship a build that keeps class names, or point me at a property I can count on the minified objects, I will count terminals directly on 2.55.2.

2.55.1 gives each disposed WebGL renderer's context and canvas back immediately, instead of waiting for GC. Dead contexts had been piling up and counting toward Chromium's cap.
2.55.2 works around addon-webgl 0.19.0's cursor-blink manager. Its interval survived dispose() and kept the renderer and its whole Terminal, scrollback included, alive. In your snapshot, that is the 20-odd renderers above the terminal count. The growth also happened under load because every hide, park or close of a focused terminal leaked one.

On the middle row (+442 MB with CPU sampling at zero): I can't explain it from hourly samples either, and I won't claim the fixes cover it. One candidate: agents or long builds writing output in background panes overnight. Each terminal keeps up to 5000 lines of scrollback by default, and a buffer that fills up grows until it reaches that limit. That growth is bounded and stops on its own, the way your 05:14-12:14 window flattens out. A leak would keep going.

Counting terminals directly is possible as of 2.57.0 (out now):

In a heap snapshot, filter constructors by WmuxLive. WmuxLiveTerminal counts the xterm terminals still reachable, and WmuxLiveWebglRenderer counts the WebGL renderers. The markers are named through property keys, which minification does not rename, and I checked that both names are in the shipped bundle.
Without a snapshot, run __wmux_terminalCensus() in the DevTools console. It returns created, disposed, collected, open and disposedButAlive for terminals and renderers. Force a GC first (the bin icon in the Memory tab). After that, disposedButAlive should be 0, and terminals.open should equal the number of terminals you have open.

I'm closing this as fixed by 2.55.1 and 2.55.2. If a 2.57.0 run shows disposedButAlive above zero after a forced GC, or WmuxLiveTerminal above your open-terminal count, please post it here and I'll reopen right away. Those two numbers are now a direct test, no proxies needed.

123 hours uninterrupted on 2.58.0, which carries the #433 fix. The climb is still there. 222 MB at start, 4239 MB now. I am sorry to bring you this rather than the clean result we were both hoping for.

hours	2.51.0	2.58.0
0	262 MB	222 MB
12	981 MB	167 MB
24	1159 MB	192 MB
36	1222 MB	735 MB
48	1714 MB	790 MB
60	1934 MB	1077 MB
72	691 MB	1493 MB
84	1800 MB	2984 MB
96	2262 MB	3206 MB
108	2317 MB	3782 MB
123	2529 MB	4239 MB

Read the gap at the bottom with care, because two things make it unfair to 2.58.0. First, the 2.51.0 row at hour 72 is not the renderer releasing anything: that is me closing two workspaces by hand while measuring. That run got a 1.2 GB reset halfway that the 2.58.0 run never got. Second, the two runs are not load-matched. So I would not read "68 % worse" out of this. What I would read is that 4239 MB after five days, still climbing, is not a fixed leak.

The first 24 hours were genuinely different

192 MB at hour 24 is better than any build I have measured, by a wide margin - 2.47.0 was at 502 MB and 2.51.0 at 1159 MB at the same age. Something your fix did is real and visible. Then it takes off, and the takeoff has a shape worth your attention:

time	hours	private	node CPU	node procs
16:14	25.5	187 MB	0 %	17
17:14	26.5	227 MB	14 %	26
18:14	27.5	303 MB	9 %	40
19:14	28.5	461 MB	2 %	24
20:14	29.5	503 MB	0 %	20
21:14	30.5	580 MB	0 %	20
22:14	31.5	664 MB	0 %	20

There is a work burst at 17:14-18:14, and then the growth continues for four more hours after the work stops: 303 to 664 MB at zero sampled load with the process count back down. That does not match "grows while you work and stays flat when wmux sits idle". Something keeps accumulating after the activity that triggered it has ended.

My load column is one instantaneous sample an hour and misses work easily, so treat it as weak evidence. But the same shape repeats later in the run.

What I owe you

I never ran __wmux_terminalCensus(). You built that for this issue and I did not get it used - the console call needs a human at the keyboard and it kept not happening across five days and four restarts. That is on me, and it means I am handing you a curve with no mechanism behind it, which is exactly the weak instrument that misled me twice already in this thread.

I will get the census run on the current process before it is restarted, and post created/disposed/garbage-collected plus disposedButAlive after a forced GC, next to the number of terminals open. If there is anything else you would rather have from a live 123-hour renderer while it is still up, say so quickly and I will take that too.

Navigation Menu

Repository navigation

Renderer private bytes reach 6.2 GB after 92 h, and are still climbing on 2.51.0 (follow-up to #385) #421

2.47.0, sampled by hand

2.51.0, sampled hourly

Why I am filing this now rather than waiting

A correction to what I told you on #385

Ruled out

What I could not do

Environment

1. JS heap next to private bytes

2. The console says it outright

3. Visible panes

Still to come

Canvas backing store

Object counts

Detached DOM

What I would take from this

The snapshot was taken with DevTools open, and DevTools retains DOM nodes

The JS-side retention, which is the same for all four and is not a DevTools artifact

On 2.55.1

The load/idle split matches your description

Renderer count from the peak snapshot

Next

The first 24 hours were genuinely different

What I owe you

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Renderer private bytes reach 6.2 GB after 92 h, and are still climbing on 2.51.0 (follow-up to #385) #421

Description

2.47.0, sampled by hand

2.51.0, sampled hourly

Why I am filing this now rather than waiting

A correction to what I told you on #385

Ruled out

What I could not do

Environment

Activity

amirlehmam commented on Nov 1, 2026

MarcusJenshaug1 commented on Nov 2, 2026

amirlehmam commented on Nov 3, 2026

MarcusJenshaug1 commented on Nov 3, 2026

1. JS heap next to private bytes

2. The console says it outright

3. Visible panes

Still to come

MarcusJenshaug1 commented on Nov 3, 2026

Canvas backing store

Object counts

Detached DOM

What I would take from this

MarcusJenshaug1 commented on Nov 3, 2026

MarcusJenshaug1 commented on Nov 3, 2026

MarcusJenshaug1 commented on Nov 3, 2026

amirlehmam commented on Nov 3, 2026

MarcusJenshaug1 commented on Nov 3, 2026

The snapshot was taken with DevTools open, and DevTools retains DOM nodes

The JS-side retention, which is the same for all four and is not a DevTools artifact

On 2.55.1

amirlehmam commented on Nov 4, 2026

MarcusJenshaug1 commented on Nov 4, 2026

The load/idle split matches your description

Renderer count from the peak snapshot

Next

amirlehmam commented on Nov 4, 2026

MarcusJenshaug1 commented on Nov 11, 2026

The first 24 hours were genuinely different

What I owe you

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions