Repository navigation
feat: security remediation, Linux (Hyprland/GNOME/KDE) port, custom OpenAI-compatible provider, agent autonomy + workflow recipes - #8
Merged
Conversation
…er + agent autonomy + workflow recipes SECURITY (docs/SECURITY-REMEDIATION.md): - WS token auth on 0.0.0.0:9001 (loopback trusted, BLINKY_REMOTE_TOKEN for remote) - Mobile sends auth token; strict CSP in tauri.conf; send_keys sanitized - ollama think:false + JSON fence tolerance; git hygiene (artifacts untracked) LINUX PORT (docs/LINUX-PORT-ROADMAP.md, phases 1-7): - backend/ layer: abc contract + hyprland impl (capture, input, window, apps, vision, portal, system profile) — DE-agnostic core, Hyprland-first - MCP layer removed; linux_mcp.py + wayland_vision.py are compat shims - tesseract OEM1/PSM3 tuned; .desktop registry + gio launch for apps CUSTOM PROVIDER (OpenAI-compatible, Zen/Go verified live): - ai/custom_client.py: text + vision (base64 image_url), thinking-model support (thinking disabled), balanced-object JSON extraction - Rust .env persistence + frontend UI (both windows) for URL/model/key AGENT AUTONOMY: - Cost-hierarchy prompt (open_app > list_windows > mouse > click > type > screenshot) - Focus-follows-mouse knowledge injected via runtime system profile - press_key alias map (enter->Return etc); open_app verified via list_windows - Compound "open X and Y" routes to agent loop; locator no longer hijacks open - click_element OCR name resolution (phrase merging, backend click) WORKFLOW RECIPES: - recipes.py: sanitized goal-path extraction, pending save + user confirmation (frontend chip), intent-aware matching, numbered inspiration context - Tauri confirm_recipe_save command + blinky://recipe-prompt event
…-SPI) GNOME/KDE implementation of the ComputerUseBackend contract — same ABC, auto- detected by DE. Correct-by-construction against standard mechanisms (untestable on Hyprland; verified by graceful degradation + full regression). NEW: - backend/atspi.py — AT-SPI a11y tree bridge: get_elements (UIElement boxes), get_active_window/list_windows (aggregate app bounds), is_available gate. Shared element/window source for GNOME + KDE; degrades to []/None when the registry is empty (Hyprland). Requires python-atspi pkg. - backend/gnome.py — GnomeBackend: portal capture, AT-SPI windows/elements, shared ydotool/wtype input, Shell.Eval focus. Gates on GNOME DBus ping. - backend/kde.py — KdeBackend: grim→portal capture, AT-SPI windows/elements, KWin Scripting.loadScript focus. Gates on KWin DBus ping. CHANGED: - backend/__init__.py — get_backend() auto-detects DE (hyprland/gnome/kde via system profile + env fallback); get_backend_name() exposes it. - backend/input.py — the only compositor-dependent pieces (cursor pos, move_to) dispatch per availability: xdotool (X11/XWayland) > hyprctl > ydotool REL w/ accel compensation. focus_window tries hyprctl → xdotool → portal activate. - backend/portal.py — added activate_window() per-DE DBus focus path. - linux_mcp_compat.get_app_state — pulls AT-SPI element tree when available. Verified on Hyprland: auto-detect → HyprlandBackend; AT-SPI degrades cleanly (0 elements, no crash); input dispatch compiled + move_to pixel-exact; python 235 passed / 10 pre-existing (no regressions); cargo check clean.
…ture
The log.txt failure ("open youtube in firefox" → 0 windows, no monitor
geometry, "Screenshot captured (no image available)") had two root causes,
both now fixed:
1. STALE HYPRLAND_INSTANCE_SIGNATURE after a session/gdm restart. The daemon
inherited the pre-restart instance; every hyprctl call silently hit a dead
socket → 0 windows, no monitor geometry → grim capture dead → the whole
vision/autopilot loop went blind (LLM got "Screenshot captured (no image
available)" and hallucinated an empty desktop).
- backend/window.py: _current_instance_sig() scans $XDG_RUNTIME_DIR/hypr/
and resolves the LIVE instance; _run_hyprctl tries the resolved sig and
retries the newest instance on connection failure. Never trusts inherited
env blindly.
- backend/input.py: _cursor_pos / move_to route hyprctl through the same
instance resolution so cursor control survives restarts too.
2. HARD-CODED grim everywhere (regression from the MCP-removal pass).
capture_screenshot() + both screenshot_tool fallback paths called
GrimFullscreenCaptureStrategy directly, which REQUIRES hyprctl monitor
geometry — no portal / X11 ImageGrab fallback. Now all three use
CaptureStrategyFactory.get_strategy() (grim → window-crop → portal →
ImageGrab), so a stale socket or missing grim can't blind the loop.
3. wake_word.py: openwakeword 0.4 API drift — download_models() removed
(now guarded) and Model(wakeword_models=...) → Model(wakeword_model_paths=)
via signature introspection. Verified: "Model loaded successfully.
Listening for wake word..." (CUDA provider warning is benign CPU fallback).
Verified: stale-env simulation now yields windows=2, monitors=1, capture with
real image + 126 OCR items (was 0/0/None); pytest 235 passed / 10 pre-existing
(no regressions).
…al, AT-SPI windows log2.txt (real GNOME run) showed every shared tool dead-ending on GNOME: - wtype: "Compositor does not support the virtual keyboard protocol" (wtype is wlroots-only; GNOME rejects it) - grim: "No monitor geometry" (get_monitors() is hyprctl-bound) - portal: "Screen capture permission denied" (non-interactive Screenshot is flat-out denied by xdg-desktop-portal-gnome) - list_windows: 0 (hyprctl only) Result: the LLM got "Screenshot captured (no image available)" + empty desktop and hallucinated, burning all 10 iterations. Fixes (backend, shared across Hyprland/GNOME/KDE): - input.py: type_text now wtype-first, ydotool type fallback on any wtype failure (uinput, works on every compositor). key() likewise: wtype first, then a ydotool raw-keycode dance (QWERTY linear-scan map: enter=28, ctrl=29, l=38, etc.). move_to gains DE-aware ordering: hyprctl first ONLY on a live Hyprland instance, xdotool for non-Hyprland X11/XWayland, ydotool REL last (prevents xdotool moving the wrong pointer on Hyprland XWayland). - window.py: _is_hyprland() now keys on a LIVE instance socket / XDG_CURRENT_ DESKTOP advertisement, NOT binary presence (hyprctl exists even under GNOME). get_active_window/list_windows route to AT-SPI on non-Hyprland DEs. - portal.py + capture.py: portal Screenshot now interactive=True so GNOME/KDE show the user-consent dialog (previously always denied). - capture_screenshot + screenshot_tool fallbacks now use the compositor-aware CaptureStrategyFactory (grim → window-crop → portal → X11 grab), never hard-coded grim. - tools.py: _detect_desktop_session derives from the OS session, not grim availability (grim absent no longer mislabels GNOME as 'x11'). Tests updated for the new factory/mock surfaces; 235 passed / 10 pre-existing (no regressions). Hyprland live-verified pixel-exact move_to + capture intact. Still needed per-DE (deferred, sudo): python-atspi (AT-SPI tree), xdotool (absolute cursor under GNOME/KDE XWayland) — both degrade gracefully.
…pture We're on a real GNOME session now (the actual Phase 8 test), which exposed: 1. _is_hyprland() wrongly returned True on GNOME: it keyed on a leftover HYPRLAND_INSTANCE_SIGNATURE socket dir still on disk from the old session. Signal of record is XDG_CURRENT_DESKTOP; socket files/binary presence are not signals. This alone sent list_windows to 0 everywhere on GNOME. 2. AT-SPI unusable in-process: Arch python-atspi ships cpython-314 bindings, but the Blinky venv is 3.12 — pyatspi import fails silently. Added a python3.14 helper subprocess (same pattern as the portal CLI helper) that queries desktop/windows/elements via AT-SPI and returns JSON. Tolerates pyatspi 2.58's snake_case get_extents(). Verified live: 2 windows (gnome-shell, FDM) + 14 elements on this GNOME session. python-atspi + python3.14 are now required (Arch system pkgs). 3. Capture dead on GNOME: grim is wlroots-only (never works on Mutter), and headless XDG portal Screenshot is auto-denied by xdg-desktop-portal-gnome (no visible parent window → no consent dialog → immediate Permit). Added DE-aware factory: skip grim on GNOME/KDE, add GnomeScreenshotCaptureStrategy (native consent flow) + _SpectacleCaptureStrategy (KDE). gnome-screenshot is the remaining install the user runs. 4. Portal now needs dbus-python + pygobject in the venv for the GLib mainloop (added; previously ImportError masqueraded as PermissionDenied). Verified live on GNOME: is_hyprland=False, AT-SPI windows=2 + elements via helper, input via ydotool fallback. pytest 235 passed / 10 pre-existing (no regressions); cargo clean.
GNOME 50 gates org.gnome.Shell.Screenshot and auto-denies headless portal Screenshot requests, so gnome-screenshot fell back to XWayland (black frames). Adds screen_cast_capture(): full ScreenCast session handshake (CreateSession → SelectSources → Start) → PipeWire node id → one gst-launch frame. Uses the calling interpreter (dbus-python + PyGObject present in the venv), not /usr/bin/python3. Still needs live GNOME consent; gates on DE in the capture factory alongside GnomeScreenshotCaptureStrategy.
Contributor
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (2)
📒 Files selected for processing (66)
📝 WalkthroughWalkthroughThis change adds custom OpenAI-compatible providers, authenticated mobile-to-desktop WebSockets, a Linux desktop automation backend, staged workflow saving, updated computer-use routing, and related security, compatibility, documentation, and validation updates. ChangesBlinky platform updates
Estimated code review effort: 5 (Critical) | ~120 minutes Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant MobileApp
participant DesktopWebSocket
participant ComputerUseLoop
participant LinuxBackend
participant RecipeRegistry
participant Frontend
MobileApp->>DesktopWebSocket: connect with remote token
DesktopWebSocket-->>MobileApp: authenticated session
ComputerUseLoop->>LinuxBackend: execute computer-use action
LinuxBackend-->>ComputerUseLoop: action result
ComputerUseLoop->>RecipeRegistry: stage successful workflow
ComputerUseLoop->>DesktopWebSocket: emit recipe_prompt
DesktopWebSocket->>Frontend: forward recipe prompt
Frontend->>RecipeRegistry: confirm or discard recipe
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning Some tools did not complete. Review the errors below. 🔧 Clippy (1.97.1)Clippy execution timed out Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Six commits of hardening + the full Linux desktop-automation port, rebased on current
main. Verified: 243 pytest passes (blocked only by 10 pre-existing env/network-dependent tests), cargo check clean, live-tested on Hyprland and live field-tested on GNOME.Highlights
🔒 Security remediation (
docs/SECURITY-REMEDIATION.md)BLINKY_REMOTE_TOKEN, auto-generated, constant-time compare); mobile sendsauth:frametauri.conf.json;send_keysmetachar sanitization; ollamathink:false+ JSON fence tolerance; git hygiene (runtime artifacts untracked)🐧 Linux port — Hyprland-first, DE-agnostic (
docs/LINUX-PORT-ROADMAP.md, phases 1–8)linux/python/backend/: ABC contract +HyprlandBackend,GnomeBackend,KdeBackend— auto-selected by DE; grim/portal/ScreenCast capture cascade, ydotool/wtype uinput input (with GNOME-compatible fallbacks), AT-SPI element trees,.desktopapp registry🔌 Custom OpenAI-compatible provider
ai/custom_client.py: text + vision via any OpenAI-compatible endpoint (Zen, OpenRouter, vLLM, LM Studio), thinking-model support, balanced-object JSON extraction.envpersistence🤖 Agent autonomy + workflow memory
press_keyalias map, cheap list_windows verification of launches, compound-instruction routingVerification
Summary by CodeRabbit
New Features
Bug Fixes