Repository navigation
Show and control the agent browser page in wmux - #455
Conversation
…rowser pane Agent mode used to show only the agent-browser dashboard. The pane now streams the Chrome page the agent drives, with native mouse, keyboard and IME input, navigation and tab controls, and a menu for a new tab or the dashboard. Viewer actions and `wmux browser` commands go through one guarded queue per session. After an uncertain command the session is re-checked before it resumes, and the command is never replayed. A paused page offers Reconnect / Continue or Start new session. The surface owns Chrome across split-tree remounts, and new sessions are captured at 2x for sharpness.
|
Thanks for this. The Page view is a real improvement over the embedded dashboard, and the code follows the agent-browser rules in AGENTS.md (single registry and daemon,
Smaller points: Happy to merge once 1–4 are addressed. The rest of the batch went out in 2.62.0. |
…launch Before agent mode starts Chrome, wmux now does these steps: 1. wmux reads the agent-browser version. If the version is older than 0.34.0, wmux stops the start and shows the update command. Version 0.34.0 is the first version that has the --pin-tab option. 2. wmux waits until the startup procedure that closes old sessions is complete. 3. wmux asks agent-browser if a daemon has the session name. This command does not start a daemon. 4. If a daemon has the session name, wmux closes the daemon. Then wmux makes sure that the daemon stopped before it starts Chrome. An earlier wmux run can leave a daemon open. Before this change, the new start used that daemon. The daemon kept its old stream port, and the control check stopped the start. On agent-browser 0.35, the daemon also did not use the launch arguments. If the startup procedure closed the daemon during the start, the start failed. Tested with agent-browser 0.34.0, 0.35.2, 0.41.2 and 0.43.0.
Before this change, mouse moves on the page view filled the command queue during a slow native command. The viewer then stopped all input, and agent commands failed with "Browser control queue is full". Now the viewer does these steps: 1. If a move is in the queue and did not start, a new move replaces its position. A new wheel event adds its deltas to a queued wheel event. Other input, page actions, tab changes and visibility changes stop this merge, so the order of input does not change. 2. The viewer accepts a maximum of 64 queued jobs for each session. If the queue is full, the viewer ignores new input, and page actions show "The page is busy". The viewer does not stop. 3. If the viewer accepted a key press or a mouse press, it always accepts the release. Thus no key or button stays down. The control queue does not count viewer jobs in its limit of 256 agent jobs. Thus viewer input cannot block an agent command. Tested with agent-browser 0.41.2: 1000 moves, 20 wheel events and 10 characters during a 5 s agent command gave 51 queued jobs. No errors occurred, and an agent command after the input was successful.
Before this change, main acknowledged each native frame on receipt. Frames continued to arrive when the renderer was slow or the page view was hidden. Now main waits for the renderer to draw or drop delivered frames before it acknowledges the native stream. While the page view is hidden, main holds the acknowledgement. URL and tab updates still pass, and agent commands continue to work. When the view returns, frame delivery resumes. Navigation reported while the view is hidden clears the cached image, so an old image cannot enable input on the new page. The renderer also acknowledges dropped old-tab frames, so they cannot stop delivery. The optional Dashboard stream and native browser capture do not change.
The short wmux deadlines could stop the native CLI before it returned a result. A command error could also disconnect a healthy page stream. Use a 35-second hard watchdog instead of short action deadlines. Give the wmux CLI 45 seconds by default, with larger budgets for explicit long waits. Treat a completed native timeout as a command error, not a lost response. Keep the original stream visible when an unknown outcome blocks input. Disconnect it only when the stream fails or browser identity cannot be trusted. Keep deadline failures during identity checks distinct from an observed identity change. Show command errors in a small notice instead of covering the live page. Keep the existing input, frame and session checks in place.
Treat legacy native read failures as unknown command outcomes. Keep the live stream visible and block new actions until explicit recovery. Apply the same check to plain startup errors. Do not retry the launch.
|
…ge view (#455) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks, that covers all four. I checked each one in the code and merged it on top of current master. The full suite passes there (7,049 tests).
It shipped in 2.64.0. Thanks for the careful follow-up. |
Problem
In Agent mode the tab embeds agent-browser's dashboard, a separate web app with its own controls.
This dashboard has many useless features, like always-enabled chat with anthropic api keys only support.
wmux's own toolbar barely drives the session: back/forward are disabled, reload reloads the dashboard rather than the page, and there are no tab controls.
wmux browserruns uncoordinated native agent-browser processes; undetected timeouts or Chrome restarts let subsequent commands silently target replaced browsers.Changes
wmux browsercommands together per session. Check daemon identity, ports and CDP endpoint before/after each command. Uncertain outcomes (timeout,outcome_unknown, browser replaced) pause the session without replay; timed-out reads (wait,snapshot,get text,screenshot) fail without pausing.wmux browsercommands through the same startup the tab uses, instead of only allocating a registry entry.--timeoutto nativewait <ref>and remove the dropped-timeout warning. The serial agent-browser daemon's 25s default wait blocked all later commands.data:URLs for Agent start/navigation; other unsupported start schemes open a blank page rather than fail.agentBrowser.currentUrl/agentBrowser.openIPC; stream page events and viewer actions replace them.Limitations
waitonly matches CSS selectors, not snapshot refs. In Agent mode,wmux browser wait <ref>still fails after its timeout, without blocking later commands or pausing the session.alert,confirm,prompt); neither viewer norwmux browserCLI can answer them. With agent-browser 0.41.2, open dialogs block later commands until nativeagent-browser dialog accept|dismisshandles them. Same limitation exists with dashboard view.Testing
npm run typecheckandgit diff --checkpassed.open,snapshot,fill,type,click,get-text,eval,screenshot,forward,reloadworks against the visible page.wait e3 1000fails with "Wait timed out after 1000ms"; subsequent commands run without delay or pause.data:page restores it.Screenshot
New default page view for agent
Still has ability to switch into dashboard view