8000 GitHub - FeV-06/Blinky: Don't know how to use the software? Just tell and in a blink know it! · GitHub
[go: up one dir, main page]
Skip to content
FeV-06
Blinky
Public
forked from KingSahil/Blinky

About

Don't know how to use the software? Just tell and in a blink know it!

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository
BranchesTags
 
 

Repository files navigation

🧠 Blinky — AI Desktop Tutor for Students

Ask. Learn. Click. Done.



Platform License

An AI-powered Windows desktop tutor that teaches users software directly on their screen using local AI.

⚡ Quick Start

1️⃣ Install Prerequisites

Install the following software:

  • Bun 1.3+
  • Rust Stable
  • Python 3.11+
  • Ollama

2️⃣ Pull the AI Model

ollama pull gemma4:e4b

3️⃣ Install Dependencies

bun install
bun run setup:python
bun run check:ollama

4️⃣ Start Blinky

bun run dev

⌨️ Open Blinky

Main Hotkey

CTRL + SHIFT + SPACE

Fallback Hotkey

CTRL + SHIFT + ENTER

Ask something like:

How do I install Python extension?

Blinky will:

  • Capture the current screen
  • Run OCR
  • Detect the active application
  • Generate AI instructions
  • Highlight matching UI elements

Provider Configuration

By default Blinky uses Ollama. To switch to Groq with image understanding, set these environment variables before running:

$env:BLINKY_AI_PROVIDER="groq"
$env:GROQ_API_KEY="your-groq-api-key"

Optional overrides:

$env:BLINKY_GROQ_MODEL="llama-3.2-90b-vision-preview"
$env:BLINKY_GROQ_URL="https://api.groq.com/openai/v1/chat/completions"

For Ollama overrides:

$env:BLINKY_AI_PROVIDER="ollama"
$env:BLINKY_OLLAMA_MODEL="gemma4:e4b"
$env:BLINKY_OLLAMA_URL="http://localhost:11434/api/generate"

🚀 What is Blinky?

Blinky is a hackathon-ready AI desktop tutor that helps students learn software in real time.

Instead of:

  • Watching long YouTube tutorials
  • Reading confusing documentation
  • Switching tabs repeatedly

Users can simply ask:

"How do I install Python extension?"
"How do I crop an image?"
"How do I export this?"

Blinky will:

  1. Capture the current screen
  2. Read visible UI text
  3. Detect the active application
  4. Generate AI instructions
  5. Highlight the exact button/menu to click

✨ Features

🌟 Recent Enhancements

🛡️ Dynamic Capture Exclusion (Flicker-Free Mode)

Blinky now uses the native Windows API (SetWindowDisplayAffinity / WDA_EXCLUDEFROMCAPTURE) to exclude its own command and overlay windows from screen captures programmatically.

  • The Blinky UI remains fully visible and active to you.
  • The screenshots captured for the AI model are completely clean, hiding the Blinky UI from its own vision without needing to minimize or hide the app.
  • Manual user screenshots (e.g., Ctrl + Win + S / Win + Shift + S) still capture Blinky correctly because capture exclusion is dynamically restored immediately after the AI's screenshot is captured (under 100ms).

🎯 Full-Width Search & Input Highlighting

Highlight boxes for search bars and text inputs are no longer constrained or shrunk to specific OCR words.

  • Blinky automatically detects when OCR text lies within a native text-input control (using UIA boundaries).
  • It scales and extends the highlight overlay to cover the entire width of the input field, providing a clean, clear visual guide.

📋 Robust Action Guides & Fallbacks

Action-oriented tasks (such as searching, downloading, or configuring settings) will always generate a step-by-step Action Guide, even when the target view, panel, or extension marketplace is currently closed.

  • Instead of defaulting to a plain text summary, Blinky guides you to open the appropriate panel or sidebar view first, followed by the search and interaction steps.
  • Non-visible targets are listed as text guidance with target_text: "" to keep guidance clear without drawing empty highlights.

⚡ Local Inference Performance Optimizations

Local Ollama (Gemma) execution speed has been optimized to 5-7 seconds (down from 15+ seconds) through:

  • Duplicate Capture Elimination: Removed redundant screenshot and OCR execution loops in the Python worker.
  • Prompt Compression: Compressed OCR layout tokens by converting items to a compact coordinate string representation, reducing prompt tokens by ~1800.
  • Timeout Tuning: Extended connection timeouts to 120 seconds to prevent local model load-time failures.

🖥️ Real-Time Screen Capture

Captures the active screen instantly when the user asks a question.

🔍 OCR-Based UI Understanding

Extracts visible text, buttons, menus, and labels from applications.

  • Windows OCR API (primary)
  • EasyOCR fallback

🧠 Local AI Reasoning

Runs fully offline using:

  • Ollama
  • gemma4:e4b

🎯 Smart Overlay Highlighting

Highlights buttons and menus directly on the user's screen.

⚡ Global Hotkey Workflow

Open Blinky instantly using:

CTRL + SHIFT + SPACE

🔒 Privacy Friendly

  • Fully local processing
  • No cloud screenshots
  • No tracking
  • No mandatory external APIs

🎯 Why It Matters

Students waste hours learning basic software workflows.

Blinky transforms software learning into an interactive real-time experience.

Benefits

✅ Learn directly inside apps
✅ No long tutorials
✅ No cloud dependency
✅ Beginner-friendly guidance
✅ Privacy-first local AI
✅ Fast workflow assistance

🏗️ Architecture

┌─────────────────────┐
│ User Question       │
└──────────┬──────────┘
           ↓
┌─────────────────────┐
│ Global Hotkey       │
└──────────┬──────────┘
           ↓
┌─────────────────────┐
│ Screen Capture      │
│ dxcam               │
└──────────┬──────────┘
           ↓
┌─────────────────────┐
│ OCR Extraction      │
│ Windows OCR         │
│ EasyOCR Fallback    │
└──────────┬──────────┘
           ↓
┌─────────────────────┐
│ Active Window       │
│ pywinauto           │
└──────────┬──────────┘
           ↓
┌─────────────────────┐
│ Ollama + Gemma      │
│ AI Step Generation  │
└──────────┬──────────┘
           ↓
┌─────────────────────┐
│ JSON Instructions   │
└──────────┬──────────┘
           ↓
┌─────────────────────┐
│ Overlay Highlight   │
│ Guidance            │
└─────────────────────┘

🛠️ Tech Stack

Component Technology
Desktop Framework Tauri 2
Frontend React + TypeScript
JavaScript Runtime / Package Manager Bun
Backend Runtime Python 3.11+
AI Runtime Ollama
AI Model gemma4:e4b
OCR Windows OCR API
OCR Fallback EasyOCR
Screen Capture dxcam
Window Detection pywinauto
Overlay System Transparent Tauri Window

📂 Project Structure

src-tauri/
├── Tauri desktop shell
├── Overlay window
└── Global hotkeys

frontend/
├── React UI
├── Overlay rendering
└── Chat interface

python/
├── Capture scripts
├── OCR pipeline
├── AI integration
├── Window detection
└── Matching logic

shared/
├── Shared schemas
└── JSON payloads

scripts/
├── Setup scripts
└── Startup helpers

⚡ Installation

1️⃣ Install Requirements

Required Software

  • Bun 1.3+
  • Rust Stable
  • Python 3.11+
  • Ollama

2️⃣ Pull the Local AI Model

ollama pull gemma4:e4b

3️⃣ Install Dependencies