8000 feat(webdriver): W3C Actions API (key, pointer, wheel) by adeelahmad · Pull Request #13 · adeelahmad/Tether · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
browser's cookies. HTTPS via a local CA (`/tether/v1/ca.pem`).
- W3C WebDriver endpoints for the common Selenium command set
(see `docs/WEBDRIVER.md`), sharing the same sessions as the proxy.
- WebDriver Actions API (`POST`/`DELETE /session/{id}/actions`): key,
mouse, pen, touch and wheel input as real CDP events, so `ActionChains`
hover, drag and drop, double/context click, modifier chords and scrolling
work.
- Control API under `/tether/v1/`: health, tab list/create/close, and
JavaScript evaluation via `Runtime.evaluate`.
- Connects to an existing Chrome (`--chrome auto|port|host:port|ws://…|DevToolsActivePort`)
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -229,12 +229,12 @@ local CA, and it only drives Chromium-family browsers.
## Status & roadmap

- [x] Tab-session core, HTTP/HTTPS proxy, WebDriver common set, control API
- [x] WebDriver Actions: hover, drag and drop, key chords, wheel, multi-touch
- [x] Safe-by-default exposure, token auth, domain allowlist seam
- [ ] MCP server face (tools over the same core) — stubbed at `/mcp`
- [ ] Remote-browser UI with live screencast and input — stubbed at `/remote`
- [x] Per-token and per-session domain policies, enforced in the browser ([docs/POLICIES.md](docs/POLICIES.md))
- [x] Per-tab console and browser log capture (control API + WebDriver `get_log`)
- [ ] WebDriver Actions

## Contributing

Expand Down
44 changes: 43 additions & 1 deletion docs/WEBDRIVER.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ With a token: `ClientConfig(remote_server_addr=..., auth_type=AuthType.BEARER, t
| Get Window Handle(s) | `GET /session/{id}/window`, `/window/h 10BC8 andles` |
| Switch To Window (own handle only) | `POST /session/{id}/window` |
| Close Window (closes the tab, ends the session) | `DELETE /session/{id}/window` |
| Perform / Release Actions | `POST`, `DELETE /session/{id}/actions` — key, pointer (mouse, pen, touch) and wheel input |
| Get Log / Get Log Types (Selenium extension) | `POST /session/{id}/se/log`, `GET …/se/log/types` (also `/log`, `/log/types`) — types `browser`, `driver` |
| Get / Set Window Rect, Maximize | `GET`/`POST …/window/rect`, `POST …/window/maximize` — reports the rect; resizing your real window is intentionally a no-op |

Expand Down Expand Up @@ -92,9 +93,50 @@ Behaviour notes:
- **Cookies** are your browser's real cookies. `DELETE /cookie` deletes the
current site's cookies from your profile — that *will* log you out.

## Actions

`ActionChains` and the W3C Actions API work: hover, drag and drop,
double and context click, modifier chords, held keys, wheel scrolling,
and multi-touch. Every action becomes a real CDP input event, so pages see
trusted `mousedown`/`keydown`/`wheel`/`touchstart` events, not synthetic
ones.

```python
from selenium.webdriver import ActionChains, Keys

ActionChains(driver).key_down(Keys.CONTROL).click(link).key_up(Keys.CONTROL).perform()
ActionChains(driver).drag_and_drop(card, column).perform()
ActionChains(driver).scroll_by_amount(0, 800).perform()
```

- **Sources**: `key`, `pointer` (`pointerType` `mouse`, `pen` or `touch`),
`wheel` and `none`, with every action type the spec defines (`pause`,
`keyDown`/`keyUp`, `pointerDown`/`pointerUp`/`pointerMove`/`pointerCancel`,
`scroll`). Sources run tick by tick; a tick lasts as long as its longest
`pause`, `pointerMove` or `scroll`, and moves and scrolls with a
`duration` are spread over it frame by frame.
- **Origins**: `viewport`, `pointer` and element origins. An element's
in-view centre point is used; unlike chromedriver, an element that is off
screen is scrolled into view rather than failing. A target outside the
viewport is `move target out of bounds`.
- **Keys**: all W3C key codes (modifiers left and right, F1–F12, numpad,
navigation keys). Shift applies the US layout (`a`→`A`, `1`→`!`);
Ctrl/Meta chords type no text. Modifiers held by a key source apply to
pointer and wheel events in the same session.
- **Clicks**: a press within 500 ms and 4 px of the last one on the same
button counts as a repeat click, so down/up/down/up is a double click.
Pen pressure, tilt and twist, and touch width/height, are passed through.
- **State** (held keys and buttons, pointer positions) persists across
`POST /actions` calls until `DELETE /actions`, which releases everything
still held, newest source first.
- **Touch** brings the tab to the front, because Chrome only delivers touch
input to a visible tab. Mouse, pen, keys and wheel work in a background
tab. `pointerCancel` cancels touch contacts and is a no-op for mouse and
pen.

## Not supported (yet)

Actions API (`/actions`), frames and shadow roots, alerts/prompts, new
Frames and shadow roots, alerts/prompts, new
window/tab creation, window resizing/fullscreen/minimise, file uploads,
print-to-PDF, and BiDi. These return `unknown command` (or
`unsupported operation`). Contributions welcome — they belong in
Expand Down
48 changes: 48 additions & 0 deletions internal/cdp/cdp.go
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,11 @@ type Tab interface {

Click(ctx context.Context, x, y float64) error
Key(ctx context.Context, k KeyEvent) error
// Mouse dispatches one mouse or pen event (move, press, release,
// wheel) in viewport CSS pixels.
Mouse(ctx context.Context, m MouseEvent) error
// Touch dispatches one touch event carrying every active touch point.
Touch(ctx context.Context, t TouchEvent) error

// Guard enforces check on every top-level document request the tab
// makes — navigations, redirects, link clicks, script-driven location
Expand Down Expand Up @@ -189,6 +194,49 @@ type KeyEvent struct {
Modifiers int `json:"modifiers,omitempty"`
}

10BC8 // MouseEvent is a single Input.dispatchMouseEvent.
type MouseEvent struct {
Type string `json:"type"` // mouseMoved, mousePressed, mouseReleased, mouseWheel
X float64 `json:"x"`
Y float64 `json:"y"`
Modifiers int `json:"modifiers,omitempty"`
Button string `json:"button,omitempty"` // none, left, middle, right, back, forward
Buttons int `json:"buttons,omitempty"`
ClickCount int `json:"clickCount,omitempty"`
DeltaX float64 `json:"deltaX,omitempty"`
DeltaY float64 `json:"deltaY,omitempty"`

PointerType string `json:"pointerType,omitempty"` // mouse, pen
Force float64 `json:"force,omitempty"`
TangentialPressure float64 `json:"tangentialPressure,omitempty"`
TiltX float64 `json:"tiltX,omitempty"`
TiltY float64 `json:"tiltY,omitempty"`
Twist int `json:"twist,omitempty"`
}

// TouchPoint is one finger of a TouchEvent.
type TouchPoint struct {
ID int `json:"id"`
X float64 `json:"x"`
Y float64 `json:"y"`
RadiusX float64 `json:"radiusX,omitempty"`
RadiusY float64 `json:"radiusY,omitempty"`
Force float64 `json:"force,omitempty"`
TangentialPressure float64 `json:"tangentialPressure,omitempty"`
TiltX float64 `json:"tiltX,omitempty"`
TiltY float64 `json:"tiltY,omitempty"`
Twist int `json:"twist,omitempty"`
}

// TouchEvent is a single Input.dispatchTouchEvent. Chrome diffs
// TouchPoints against the previous event, so it must list every finger
// still down; touchEnd and touchCancel carry none.
type TouchEvent struct {
Type string `json:"type"` // touchStart, touchMove, touchEnd, touchCancel
TouchPoints []TouchPoint `json:"touchPoints"`
Modifiers int `json:"modifiers,omitempty"`
}

// JSError is an exception thrown by page JavaScript.
type JSError struct {
Text string
Expand Down
52 changes: 47 additions & 5 deletions internal/cdp/cdptest/cdptest.go
Original file line number Diff line number Diff line change
Expand Up @@ -148,6 +148,13 @@ type Tab struct {
logSeq uint64
CallResult json.RawMessage
EvalResult json.RawMessage
// ScriptResult, when set, is returned as the "v" property of the
// object CallFunction yields, the shape the WebDriver script wrapper
// produces (a JSON-encoded value).
ScriptResult any
// Input records every Key, Mouse and Touch event, in order, as
// cdp.KeyEvent, cdp.MouseEvent or cdp.TouchEvent values.
Input []any

guard func(string) error
}
Expand Down Expand Up @@ -221,17 +228,35 @@ func (t *Tab) CallFunction(_ context.Context, decl string, opts cdp.CallOptions)
t.mu.Lock()
t.Calls = append(t.Calls, Call{Decl: decl, Args: opts.Args})
res := t.CallResult
script := t.ScriptResult
t.mu.Unlock()
if script != nil {
return &cdp.Remote{ObjectID: "script-result", Type: "object"}, nil
}
if res == nil {
res = json.RawMessage("null")
}
return &cdp.Remote{Type: "object", Value: res}, nil
}

func (t *Tab) Properties(context.Context, string) ([]cdp.Property, error) { return nil, nil }
func (t *Tab) Release(context.Context, string) error { return nil }
func (t *Tab) NodeID(context.Context, string) (int, error) { return 1, nil }
func (t *Tab) ResolveNode(context.Context, int) (string, error) { return "obj", nil }
func (t *Tab) Properties(_ context.Context, objectID string) ([]cdp.Property, error) {
t.mu.Lock()
script := t.ScriptResult
t.mu.Unlock()
if objectID != "script-result" || script == nil {
return nil, nil
}
inner, err := json.Marshal(script)
if err != nil {
return nil, err
}
v, _ := json.Marshal(string(inner))
return []cdp.Property{{Name: "v", Value: &cdp.Remote{Type: "string", Value: v}}}, nil
}

func (t *Tab) Release(context.Context, string) error { return nil }
func (t *Tab) NodeID(context.Context, string) (int, error) { return 1, nil }
func (t *Tab) ResolveNode(context.Context, int) (string, error) { return "obj", nil }

func (t *Tab) ResponseBody(context.Context, string) ([]byte, error) {
return t.page().Body, nil
Expand All @@ -245,7 +270,17 @@ func (t *Tab) Cookies(context.Context, []string) ([]cdp.Cookie, error) { return
func (t *Tab) SetCookie(context.Context, cdp.Cookie) error { return nil }
func (t *Tab) DeleteCookie(context.Context, string, string) error { return nil }
func (t *Tab) Click(context.Context, float64, float64) error { return nil }
func (t *Tab) Key(context.Context, cdp.KeyEvent) error { return nil }

func (t *Tab) Key(_ context.Context, k cdp.KeyEvent) error { return t.record(k) }
func (t *Tab) Mouse(_ context.Context, m cdp.MouseEvent) error { return t.record(m) }
func (t *Tab) Touch(_ context.Context, e cdp.TouchEvent) error { return t.record(e) }

func (t *Tab) record(ev any) error {
t.mu.Lock()
t.Input = append(t.Input, ev)
t.mu.Unlock()
return nil
}

// Guard records check; Navigate refuses URLs it rejects, like the real
// browser-side guard.
Expand All @@ -256,6 +291,13 @@ func (t *Tab) Guard(_ context.Context, check func(string) error) error {
return nil
}

// Events returns a copy of the recorded input events.
func (t *Tab) Events() []any {
t.mu.Lock()
defer t.mu.Unlock()
return append([]any(nil), t.Input...)
}

// Guarded reports whether a navigation guard is installed.
func (t *Tab) Guarded() bool {
t.mu.Lock()
Expand Down
25 changes: 25 additions & 0 deletions internal/cdp/tab.go
Original file line number Diff line number Diff line change
Expand Up @@ -560,3 +560,28 @@ func (t *chromeTab) Click(ctx context.Context, x, y float64) error {
func (t *chromeTab) Key(ctx context.Context, k KeyEvent) error {
return t.call(ctx, "Input.dispatchKeyEvent", k, nil)
}

func (t *chromeTab) Mouse(ctx context.Context, m MouseEvent) error {
if m.Type == "mouseWheel" { // Chrome requires both deltas, even when zero
return t.call(ctx, "Input.dispatchMouseEvent", struct {
MouseEvent
DeltaX float64 `json:"deltaX"`
DeltaY float64 `json:"deltaY"`
}{m, m.DeltaX, m.DeltaY}, nil)
}
return t.call(ctx, "Input.dispatchMouseEvent", m, nil)
}

func (t *chromeTab) Touch(ctx context.Context, e TouchEvent) error {
if e.TouchPoints == nil {
e.TouchPoints = []TouchPoint{}
}
// Chrome only acknowledges touch input in a visible tab; in a
// background tab the dispatch hangs. Mouse and keys need no such step.
if e.Type == "touchStart" {
if err := t.call(ctx, "Page.bringToFront", nil, nil); err != nil {
return err
}
}
return t.call(ctx, "Input.dispatchTouchEvent", e, nil)
}
Loading
Loading
0