Screen-aware tutoring · Voice interaction · Desktop automation · Mobile control
Learning complex desktop software often means switching between tutorials, documentation, and the application itself. Blinky brings help into the workflow: ask a question about the current screen, get a grounded next step, and see the relevant control highlighted in the app. When requested, Blinky can also carry out bounded desktop actions. A companion app extends selected controls to an Android phone.
| For | Students, developers, and anyone learning unfamiliar desktop software |
| Problem | Tutorials and documentation are disconnected from the interface being learned |
| Approach | Screen understanding, contextual guidance, visual highlights, voice, and optional automation |
| Capability | What it does |
|---|---|
| System 1 Neural Decision Engine | Sub-millisecond (<1ms) non-autoregressive ONNX Transformer classifier (decision_engine/) routing queries before LLM invocation. |
| Blinky Notebook Hub & OKF | Offline Knowledge Format (OKF) personal workspace (docs/OKF.md) with 100% full-context document reasoning, "▶ Execute on PC" actionable citations, and standalone mobile mode with Android KeyStore encryption. |
Pure Rust Agent (blinky-agent) |
Ultra-fast native Rust autonomous agent loop (crates/blinky-agent) with multi-provider LLM calling (OpenAI, Gemini, Groq, Ollama), native tool registry, and embedded SQLite vector store. |
| Screen-aware tutoring | Captures the active screen, combines OCR with accessibility or visual UI detection, and gives step-by-step guidance with on-screen highlights. |
| Voice interaction | Supports realtime speech input, voice-agent tool calls, spoken responses, turn-taking, and synchronized readback through configured voice providers. |
| Hey Blinky wake word | Runs the bundled python/hey_blinky.onnx model locally through ONNX Runtime to detect the wake phrase and start hands-free interaction. |
| Agent Mode | Uses a bounded observe-and-act loop for app launching, keyboard shortcuts, typing, scrolling, and grounded clicks. |
Linux desktop daemon (blinky-daemon) |
Native headless Rust daemon for Wayland (Hyprland/wlroots) and X11 with direct /dev/uinput, AT-SPI2 DBus trees, and PipeWire capture. |
| WhatsApp and AI summaries | Connects WhatsApp Web by QR code, reports connection status, lists chats, and creates recaps of selected group or direct conversations. |
| AiCut C++17 & MCP Video Suite | C++17 core engine, CMake build, faster-whisper forced word aligner, intelligent silence removal, and MCP server (aicut_mcp.py) for natural language video editing. |
| Web research | Uses the Web Intelligence Layer and SearXNG to retrieve pages and return synthesized answers with clickable sources. Public search fallbacks are available when configured local search is unavailable. |
| Browser and app tools | Provides persistent browser automation, appl 8000 ication launching, Spotify playback, product search, YouTube channel statistics, cryptocurrency prices, and Wikipedia lookups. |
| Android companion | Connects to the desktop over an authenticated WebSocket with TLS public-key pinning for remote chat, voice, quick actions, and PC controls over LAN, USB, or Tailscale. |
| Remote files and camera transfer | Browses PC files, transfers files between phone and computer with 16MB streaming chunks, syncs camera selections, and sends media into supported AiCut workflows. |
| PC power and Wake-on-LAN | Shows system telemetry and supports lock, sleep, hibernate, restart, shutdown, and Wake-on-LAN for appropriately configured hardware and networks. |
| Antigravity IDE bridge & HITL Gate | Streams IDE lifecycle events and terminal actions to mobile on port 9003 with human-in-the-loop mobile safety approval cards. |
| ESP32 lighting | Controls RGB color and brightness through the Blinky ESP32 daemon firmware. |
| Android access options | Integrates RevenueCat purchase and entitlement handling, with configured promo codes available for PC-control access. |
Some capabilities need API credentials, optional system packages, or connected hardware. Local Ollama inference is available; selecting a cloud AI or voice provider sends the relevant request data to that provider.
showcase.mp4 |
|||
|
|
|
|
|
flowchart LR
U(["👤 User"])
subgraph FRONT["🖥️ Frontend"]
UI["React 19 / Vite UI<br/>(Command bar, overlay)"]
end
subgraph HOST["⚙️ Desktop host"]
T["Tauri + Rust (IPC)"]
F["blinky-agent<br/>(Pure Rust Orchestrator)"]
D["blinky-daemon<br/>(Rust OS Subsystem)"]
end
subgraph BRAIN["🧠 Intelligence"]
R["System 1 Intent Classifier<br/>(<4ms Non-autoregressive)"]
AI["System 2 Reasoning<br/>(OmniRoute, Groq, Ollama)"]
end
subgraph SYSTEM["🛠️ Subsystems"]
C["Capture & Desktop<br/>(Hyprland, wlroots)"]
O["Inputs & Accessibility<br/>(uinput, AT-SPI2)"]
end
M["📱 Android companion"]
U --> UI
UI <--> T
T <--> F
M <-->|"Authenticated<br/>WebSocket"| T
F --> R
F --> AI
F <-->|"UNIX Socket (:9001)"| D
D --> C
D --> O
classDef user fill:#ea580c,stroke:#9a3412,color:#fff,stroke-width:2px
classDef front fill:#fb923c,stroke:#c2410c,color:#1c1917
classDef host fill:#f97316,stroke:#9a3412,color:#1c1917
classDef brain fill:#c2410c,stroke:#7c2d12,color:#fff
classDef system fill:#fdba74,stroke:#ea580c,color:#1c1917
classDef mobile fill:#fbbf24,stroke:#b45309,color:#1c1917
class U user
class UI front
class T,F,D host
class R,AI brain
class C,O system
class M mobile
linkStyle default stroke:#f97316,stroke-width:2px
| Layer | Main components |
|---|---|
| Desktop interface | React 19, TypeScript, Tauri 2 multi-window HUD overlay (WDA_EXCLUDEFROMCAPTURE) |
Assistant (blinky-agent) |
Pure Rust orchestrator, multi-provider AI engine (OpenAI, Gemini, Groq, Ollama), native tool registry, embedded vector store |
OS & Inputs (blinky-daemon) |
Headless Rust daemon for Hyprland/wlroots, visual capture, AT-SPI2, /dev/uinput, coordinate transformations |
Decision Engine (decision_engine/) |
Sub-millisecond (<1ms) non-autoregressive ONNX Transformer reflex classifier and calibrated logit router |
Knowledge Hub (docs/OKF.md) |
Offline Knowledge Format (OKF) context engine, actionable "▶ Execute on PC" desktop execution, Android KeyStore sync |
Media & Video (common/aicut/) |
C++17 video processing engine, CMake, FFmpeg, faster-whisper forced alignment, and AiCut MCP server |
| AI and voice | Local Ollama, Groq, AssemblyAI, Sarvam saaras:v3/bulbul:v3 with synchronized dynamic word highlight |
| Integrations & Bridges | Antigravity IDE bridge (:9003) with mobile HITL safety gate, ESP32 NeoPixel daemon, WhatsApp Web bot |
| Mobile Companion | Expo React Native with TLS public-key pinning, Sentinel telemetry, WoL, and 16MB chunked file transfer |
| Requirement | Needed for |
|---|---|
| Git | Clone the repository |
| Bun 1.3 or newer | Desktop JavaScript dependencies and development scripts |
| Python 3.11+ | Wake-word local ONNX model and decision engine training |
| Rust stable toolchain | Tauri desktop application |
| Ollama or a configured cloud AI provider | Assistant responses |
| Docker Compose | Optional local SearXNG search service |
| Android Studio/SDK and ADB | Building or connecting the Android companion |
Windows is the primary desktop setup. Linux has a separate setup path and additional desktop packages; consult the Linux guide for supported environments and requirements.
git clone https://github.com/KingSahil/Blinky.git
cd Blinky
powershell -ExecutionPolicy Bypass -File setup.ps1The setup script installs the project dependencies, prepares the Python environment, installs the Playwright browser, and creates .env from the example when needed. It does not add API keys for you.
Install Ollama and pull the configured local model if you want local inference:
ollama pull gemma4:e4bSet the AI provider and any optional provider keys in .env. Keep credentials private and do not commit .env.
Start Blinky:
bun run devTo run without starting the local Docker search service:
bun run dev:no-dockergit clone https://github.com/KingSahil/Blinky.git
cd Blinky
bun run setup:linuxInstall the system packages required by your distribution and desktop session as described in linux/quick_start.md. For local AI, install Ollama, start its service, and pull gemma4:e4b. Configure .env, then run:
bun run devLinux capabilities depend on the compositor and installed utilities. The current native implementation is Hyprland-first; see the Linux port roadmap for GNOME and KDE status.
The mobile app is in common/mobile. Start the Expo development server with:
cd common/mobile
bun install
bun run startFor a native Android build, use the provided install_apk.bat workflow or the Android distribution guide. Some native modules and RevenueCat purchase flows require a development build or standalone APK rather than the standard Expo Go client. Connect the phone and desktop through the app's pairing flow; the desktop WebSocket listener uses port 9001 by default.
The desktop development script starts SearXNG through Docker Compose when Docker is available. To start it directly:
docker compose -f common/docker-compose.yml up -d| Path | Contents |
|---|---|
crates/blinky-agent/ |
Pure Rust AI agent orchestration loop, multi-provider LLM engine, native tool registry, embedded vector store |
crates/blinky-daemon/ |
Pure Rust OS/hardware headless Linux daemon (Wayland/X11, AT-SPI2 DBus, /dev/uinput, PipeWire) |
decision_engine/ |
Sub-millisecond (<1ms) ONNX Transformer reflex classification model, dataset generator, and calibration |
common/frontend/ |
React 19 command bar, visual overlay (WDA_EXCLUDEFROMCAPTURE), voice, guidance, and autopilot UI |
common/src-tauri/ |
Tauri 2 desktop host, Rust platform services, TLS WebSocket gateway (:9001), and 16MB file streamer (:9002) |
common/python/notebooks/ |
Blinky Notebook Hub & Offline Knowledge Format (OKF) engine, document parsers, and vector store |
common/aicut/ |
C++17 video processing engine, CMake build, faster-whisper forced alignment, and AiCut MCP server |
common/mobile/ |
Expo/React Native Android companion, remote control, files, and standalone Android KeyStore support |
common/whatsapp_backend/ |
WhatsApp Web session, chat, and summary service |
windows/ |
Windows-specific Rust, setup, and system integrations |
linux/ |
Linux setup scripts and quick-start guide |
python/ |
Shared wake-word listener and bundled hey_blinky.onnx model |
esp32_firmware/ |
ESP32 universal daemon firmware for NeoPixel lighting and device control |
docs/ |
Platform, security, voice, OKF, distribution, and architecture guides |
local-docs/ |
Runtime flow, configuration, testing, and release notes |
The desktop setup creates the root .env from .env_example when needed. Configure only the providers and integrations you plan to use. Mobile RevenueCat and promo-code values belong in common/mobile/.env. Common settings include:
| Setting | Purpose |
|---|---|
BLINKY_AI_PROVIDER |
Selects the assistant model provider |
GROQ_API_KEY |
Enables Groq cloud inference |
SARVAM_API_KEY |
Enables Sarvam speech services |
ASSEMBLY_AI_API_KEY |
Enables AssemblyAI voice services |
ESP32_HOST |
Address of the ESP32 daemon |
EXPO_PUBLIC_REVENUECAT_GOOGLE_KEY |
RevenueCat Android public SDK key |
EXPO_PUBLIC_PROMO_CODES |
Configured Android promo-code access options |
Keep .env and all secret keys out of source control. The mobile desktop connection uses authentication and TLS support. Screen, microphone, and message data may be sent to the cloud services you configure; review each provider's terms and settings before enabling it. Local Ollama inference keeps model requests on the local machine.
| Guide | Covers |
|---|---|
BLINKY_SUMMARY.md |
Full architectural summary, System 1 engine, pure-Rust agent/daemon, OKF, and integrations |
docs/OKF.md |
Blinky Notebook Hub: Offline Knowledge Format (OKF) & RAG architecture plan |
linux/quick_start.md |
Linux dependencies, inference setup, OCR, and launch steps |
docs/LINUX-PORT-ROADMAP.md |
Linux backends and desktop-environment support status |
docs/ASSEMBLYAI_VOICE_AGENT.md |
AssemblyAI voice-agent integration |
docs/HERMES-INTEGRATION-PLAN.md |
Computer-use architecture and integration notes |
docs/SECURITY-REMEDIATION.md |
Security design and remediation details |
docs/REVENUECAT-AND-ANDROID-DISTRIBUTION-GUIDE.md |
Android builds, billing, entitlements, and promo codes |
common/mobile/APP-FLOW.md |
Mobile app screens and user flows |
local-docs/ARCHITECTURE.md |
Current runtime architecture |
local-docs/CONFIGURATION.md |
Environment and configuration reference |
local-docs/TESTING.md |
Existing verification and test workflows |
common/ai_docs/00_repo_summary.md |
Developer-oriented repository map and reading order |
common/ai_docs/10_aicut_video_editor.md |
AiCut video editor architecture and operations |
| Contributor | GitHub | Contribution area |
|---|---|---|
| Sparsh Khanna | @KhannaSparsh0001 | Voice integration and product interface |
| Sahil | @KingSahil | Backend and Tauri desktop application |
| FeV-06 | @FeV-06 | Mobile application and Linux development |
| meharwanfr | @meharwanfr | Linux desktop development |
| Divyam | @divi912 | Mobile file transfer and destination-aware media workflows, WSS implmentation |
| Vedant | @vedanthaha | Mobile interface and usability improvements |
| Boldbug | @boldbug1 | Automated QR pairing , setup files ,docs maintainence |
- AssemblyAI for realtime speech recognition and voice-agent services
- Sarvam AI for speech services
- Nous Research for the
cua-drivercomputer-use backend - Microsoft for OmniParser
- Tauri, React, Expo, and React Native for the application frameworks
- The Blinky contributors and users who have tested the desktop and mobile workflows
Ask. Learn. Automate. Control from anywhere.