8000 Show and control the agent browser page in wmux by strayge · Pull Request #455 · amirlehmam/wmux · GitHub
[go: up one dir, main page]

Skip to content

Show and control the agent browser page in wmux - #455

Merged
amirlehmam merged 6 commits into
amirlehmam:masterfrom
strayge:feat/agent-browser-viewer-pr
Nov 18, 2026
Merged

amirlehmam merged 6 commits into
amirlehmam:masterfrom
strayge:feat/agent-browser-viewer-pr

Conversation

@strayge
@strayge strayge commented Nov 14, 2026
Copy link
Copy Markdown
Contributor

Problem

In Agent mode the tab embeds agent-browser's dashboard, a separate web app with its own controls.
This dashboard has many useless features, like always-enabled chat with anthropic api keys only support.
wmux's own toolbar barely drives the session: back/forward are disabled, reload reloads the dashboard rather than the page, and there are no tab controls.

wmux browser runs uncoordinated native agent-browser processes; undetected timeouts or Chrome restarts let subsequent commands silently target replaced browsers.

Changes

  • Stream Chrome into the browser tab (Page view) with native mouse/keyboard/IME/wheel input. New sessions capture at 2x for sharp text.
  • Control Agent mode navigation, back/forward/reload and tabs through the address bar; the … menu opens a new tab or dashboard.
  • Queue viewer actions and wmux browser commands together per session. Check daemon identity, ports and CDP endpoint before/after each command. Uncertain outcomes (timeout, outcome_unknown, browser replaced) pause the session without replay; timed-out reads (wait, snapshot, get text, screenshot) fail without pausing.
  • Recover paused pages: Reconnect / Continue re-checks the session before resuming; Start new session closes it and opens a blank page. Exited daemons count as closed; unknown-outcome startups cannot relaunch until a new session starts.
  • Preserve surface-owned Chrome sessions across split-tree remounts; only closing the surface or switching to Web ends them.
  • Start the Agent session for wmux browser commands through the same startup the tab uses, instead of only allocating a registry entry.
  • Pass --timeout to native wait <ref> and remove the dropped-timeout warning. The serial agent-browser daemon's 25s default wait blocked all later commands.
  • Accept data: URLs for Agent start/navigation; other unsupported start schemes open a blank page rather than fail.
  • Remove the agentBrowser.currentUrl / agentBrowser.open IPC; stream page events and viewer actions replace them.
  • Add a short Agent mode README paragraph.

Limitations

  • agent-browser's wait only matches CSS selectors, not snapshot refs. In Agent mode, wmux browser wait <ref> still fails after its timeout, without blocking later commands or pausing the session.
  • Page view omits dialogs (alert, confirm, prompt); neither viewer nor wmux browser CLI can answer them. With agent-browser 0.41.2, open dialogs block later commands until native agent-browser dialog accept|dismiss handles them. Same limitation exists with dashboard view.
  • Native commands can still restart Chrome or leave duplicate daemons; wmux detects identity changes and pauses, without preventing them.

Testing

  • Full suite: 6,834 passed, 10 skipped (419 test files passed, 2 skipped).
  • npm run typecheck and git diff --check passed.
  • Agent-driven testing with agent-browser 0.41.2:
    • Page view shows the live page; double- and triple-click select text.
    • CLI sequence open, snapshot, fill, type, click, get-text, eval, screenshot, forward, reload works against the visible page.
    • wait e3 1000 fails with "Wait timed out after 1000ms"; subsequent commands run without delay or pause.
    • Switching Web → Agent on a data: page restores it.

Screenshot

New default page view for agent

image

Still has ability to switch into dashboard view

image

…rowser pane

Agent mode used to show only the agent-browser dashboard. The pane now
streams the Chrome page the agent drives, with native mouse, keyboard and
IME input, navigation and tab controls, and a menu for a new tab or the
dashboard.

Viewer actions and `wmux browser` commands go through one guarded queue
per session. After an uncertain command the session is re-checked before
it resumes, and the command is never replayed. A paused page offers
Reconnect / Continue or Start new session. The surface owns Chrome across
split-tree remounts, and new sessions are captured at 2x for sharpness.
@amirlehmam
Copy link
Copy Markdown
Owner

Thanks for this. The Page view is a real improvement over the embedded dashboard, and the code follows the agent-browser rules in AGENTS.md (single registry and daemon, isWmuxSessionName gating closes, visibility hiding, no new listening port). I held it out of 2.62.0 because of four behaviour problems that would hit unattended agents:

  1. One slow command stalls the agent until a human clicks. A mutating command that overruns its budget pauses the whole session (agent-browser-control.ts ~246/297, budgets in v2-browser.ts ~519), and every later wmux browser command is rejected. The only way out is Reconnect / Continue in the viewer; there is no CLI path. Example: wmux browser click e7 on a covered element: agent-browser retries for its 25 s default, wmux gives up at 4 s, and the agent is stuck. Before this PR a timeout failed only that command. Please make a timeout fail the command, or give the CLI a way to resume.
  2. Every frame now goes through the main process, hidden tabs included. Main parses each frame and acks it immediately (agent-browser-viewer.ts ~184/252, agent-browser-capture.ts), so agent-browser keeps streaming up to 30 fps of JPEG q95 at 2x whatever the renderer does. An animated page in a background tab means tens of MB/s parsed on the event loop that also carries terminal output. Please ack after the renderer has drawn, and stop or throttle the stream while the tab is hidden.
  3. Pointer moves share the 256-job command queue. Moving the mouse over the page during a 5 s open fills it in a few seconds: the viewer pauses ("Page input queue is full") and agent commands get "Browser control queue is full". Coalescing moves, or keeping them out of the native command queue, would fix it.
  4. It needs agent-browser ≥ 0.41 with no version check. Startup now relies on session info --json runtime fields, get cdp-url, the stream's pacing=ack, tabs/status messages and wait --timeout. On 0.35 (what I have installed) Agent mode cannot start. A minimum-version gate with a fallback to the dashboard view would cover it.

Smaller points: --force-device-scale-factor=2 makes wmux browser screenshot return 4x the pixels (more tokens for agents); the health check before and after each command costs three extra processes inside a 4 s budget; the dashboard is still started even though the Page view is now the default; and the 16 new strings exist only in en.ts.

Happy to merge once 1–4 are addressed. The rest of the batch went out in 2.62.0.

…launch

Before agent mode starts Chrome, wmux now does these steps:

1. wmux reads the agent-browser version. If the version is older than
   0.34.0, wmux stops the start and shows the update command. Version
   0.34.0 is the first version that has the --pin-tab option.
2. wmux waits until the startup procedure that closes old sessions is
   complete.
3. wmux asks agent-browser if a daemon has the session name. This command
   does not start a daemon.
4. If a daemon has the session name, wmux closes the daemon. Then wmux
   makes sure that the daemon stopped before it starts Chrome.

An earlier wmux run can leave a daemon open. Before this change, the new
start used that daemon. The daemon kept its old stream port, and the
control check stopped the start. On agent-browser 0.35, the daemon also
did not use the launch arguments. If the startup procedure closed the
daemon during the start, the start failed.

Tested with agent-browser 0.34.0, 0.35.2, 0.41.2 and 0.43.0.
Before this change, mouse moves on the page view filled the command
queue during a slow native command. The viewer then stopped all input,
and agent commands failed with "Browser control queue is full".

Now the viewer does these steps:

1. If a move is in the queue and did not start, a new move replaces
   its position. A new wheel event adds its deltas to a queued wheel
   event. Other input, page actions, tab changes and visibility changes
   stop this merge, so the order of input does not change.
2. The viewer accepts a maximum of 64 queued jobs for each session. If
   the queue is full, the viewer ignores new input, and page actions
   show "The page is busy". The viewer does not stop.
3. If the viewer accepted a key press or a mouse press, it always
   accepts the release. Thus no key or button stays down.

The control queue does not count viewer jobs in its limit of 256 agent
jobs. Thus viewer input cannot block an agent command.

Tested with agent-browser 0.41.2: 1000 moves, 20 wheel events and
10 characters during a 5 s agent command gave 51 queued jobs. No errors
occurred, and an agent command after the input was successful.
Before this change, main acknowledged each native frame on receipt. Frames
continued to arrive when the renderer was slow or the page view was hidden.

Now main waits for the renderer to draw or drop delivered frames before it
acknowledges the native stream. While the page view is hidden, main holds
the acknowledgement. URL and tab updates still pass, and agent commands
continue to work.

When the view returns, frame delivery resumes. Navigation reported while
the view is hidden clears the cached image, so an old image cannot enable
input on the new page. The renderer also acknowledges dropped old-tab
frames, so they cannot stop delivery.

The optional Dashboard stream and native browser capture do not change.
The short wmux deadlines could stop the native CLI before it returned a
result. A command error could also disconnect a healthy page stream.

Use a 35-second hard watchdog instead of short action deadlines. Give the
wmux CLI 45 seconds by default, with larger budgets for explicit long waits.
Treat a completed native timeout as a command error, not a lost response.

Keep the original stream visible when an unknown outcome blocks input.
Disconnect it only when the stream fails or browser identity cannot be
trusted. Keep deadline failures during identity checks distinct from an
observed identity change.

Show command errors in a small notice instead of covering the live page.
Keep the existing input, frame and session checks in place.
Treat legacy native read failures as unknown command outcomes.
Keep the live stream visible and block new actions until explicit recovery.
Apply the same check to plain startup errors. Do not retry the launch.
@strayge
strayge commented Nov 17, 2026
Copy link
Copy Markdown
Contributor Author
  1. Command timeouts: avoided adding a recovery CLI command because that would move wmux-specific recovery knowledge and responsibility into the agent. Instead, timeout errors no longer stop a healthy Page stream. Completed native action timeouts fail only that command.

    The 4s cutoff was replaced with a 35s hard watchdog, above agent-browser’s 30s default socket-read timeout, so it can report its own failure before wmux terminates the CLI. Ordinary action errors now show a small notice instead of disconnecting the viewer.

    Only rare cases like socket-read failure can still pause control and require explicit Reconnect, while frames keep updating.

  2. Frame backpressure: Native ACKs now wait for the renderer to confirm that it drew or dropped the frame.
    Hidden Page views withhold ACKs.
    Applies only to the Page view. The Dashboard view is unchanged and keeps streaming, as it does on master.

  3. Pointer queue saturation: Consecutive pointer moves and wheel events are now combined in the queue instead of adding a job for every event.

  4. Version and startup compatibility: Verified that the minimum version is 0.34.0. Added a version check with an upgrade message.

    One possible cause of the 0.35 startup failure is a daemon left by an earlier wmux run, retaining its old stream port. So before starting Agent mode, wmux now waits for ongoing orphan cleanup, checks for a leftover daem 8000 on with this exact session name, closes it if present, and confirms that it has stopped before launching Chrome.

@amirlehmam
amirlehmam merged commit ad75bd9 into amirlehmam:master Nov 18, 2026
amirlehmam added a commit that referenced this pull request Nov 18, 2026
…ge view (#455)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@amirlehmam
Copy link
Copy Markdown
Owner

Thanks, that covers all four. I checked each one in the code and merged it on top of current master. The full suite passes there (7,049 tests).

  1. Timeouts: a slow action now fails only that command, behind the 35 s watchdog, and the Page stream keeps going. Pausing only on a failed native read is the right line.
  2. Backpressure: the ACK now waits for the renderer to draw or drop the frame, and a hidden Page view withholds it. Keeping the Dashboard view as it was is fine.
  3. Pointer queue: merging consecutive moves and wheel steps, with any other job acting as a barrier, fixes the queue filling up.
  4. Version: the 0.34.0 floor with the upgrade hint, plus closing a leftover daemon with the same session name before launch. That leftover-daemon explanation fits what I saw on 0.35.

It shipped in 2.64.0. Thanks for the careful follow-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0