8000 mimik/AGENTS.md at dev · alahmadics/mimik · GitHub
[go: up one dir, main page]

Skip to content

Latest commit

 

History

History
247 lines (210 loc) · 13.4 KB

AGENTS.md

File metadata and controls

247 lines (210 loc) · 13.4 KB

Mimik

Open-source Chrome extension that auto-captures browser workflows and generates step-by-step guides. No backend, no account, no data leaves the browser.

What It Does

You click "Record," perform a workflow in your browser, and Mimik automatically captures each action as a step with an annotated screenshot and description. You can edit the guide, replay it on a live page, or export it as a file.

Core loop: Record → Edit → Replay or Export.

Architecture

Everything runs in the Chrome extension. No backend.

  • Storage: IndexedDB via Dexie.js (browser-local)
  • AI descriptions: optional, user provides their own API key in settings
  • Export: generated client-side (no server rendering)
  • No auth, no database, no hosting, no Docker

Directory Structure

src/
├── core/                    # Business logic (no UI dependencies)
│   ├── capture/             # Recording pipeline
│   │   ├── ai/              # AI description + title generation (Vercel AI SDK)
│   │   │   ├── description.ts   # getAIDescription (DOM context → AI → step text)
│   │   │   ├── title.ts         # generateGuideTitle (steps → AI → guide name)
│   │   │   ├── models.ts        # AI_PROVIDERS config (OpenAI/Anthropic model lists)
│   │   │   ├── prompts.ts       # Prompt templates
│   │   │   └── provider.ts      # createModel factory (OpenAI/Anthropic)
│   │   ├── dom/              # DOM extraction utilities
│   │   │   ├── context.ts       # DOMContext extraction + serialization
│   │   │   ├── element-meta.ts  # extractElementMeta (selector, text, aria, rect)
│   │   │   └── element-utils.ts # findFocusableAncestor, isTextField, etc.
│   │   ├── events/           # Event capture system
│   │   │   ├── handlers.ts      # CaptureController class + startCapture
│   │   │   └── input-session.ts # InputSession (typing lifecycle)
│   │   ├── machine.ts        # xstate capture state machine
│   │   ├── session.ts        # CaptureSession (lifecycle manager)
│   │   ├── spa-nav.ts        # SPA navigation tracking
│   │   ├── start-notification.ts # Recording notification overlay
│   │   └── step-description.ts   # Fallback rule-based descriptions
│   ├── blur/                # Smart blur: regex presets, DOM scanner, element picker, panel UI
│   ├── export/              # HTML, PDF, DOCX, Markdown, video generators + shared utils
│   └── guides/              # Data layer: types, Dexie DB, CRUD service
├── entrypoints/             # Chrome extension entry points (WXT)
│   ├── background/          # Service worker: state machine, message handlers, tab management
│   ├── content.ts           # Content script: CaptureSession, event listeners
│   ├── sidepanel/           # Side panel React mount
│   ├── fullview/            # Full-page view React mount
│   ├── onboarding/          # Onboarding wizard (opens on first install)
│   └── options/             # Settings page React mount
├── lib/                     # Shared utilities
│   ├── messaging.ts         # Extension messaging protocol (webext-core)
│   ├── port.ts              # Long-lived port: background ↔ sidepanel
│   ├── browser-api.ts       # Chrome API wrappers
│   ├── tab-messages.ts      # Content script message types
│   ├── logger.ts            # Logging utility
│   └── utils.ts             # Shared helpers (dates, URLs, cn)
├── stores/                  # Zustand state stores
│   └── fullview.ts          # Fullview UI state (search, counts, guide data)
└── ui/                      # React components
    ├── components/ui/       # shadcn/ui primitives (button, input, dialog, badge)
    ├── fullview/            # Full-page dashboard
    │   ├── components/      # Extracted sub-components (grid, list, search, etc.)
    │   ├── App.tsx
    │   ├── TopNav.tsx
    │   ├── SearchModal.tsx
    │   ├── GuideContent.tsx
    │   ├── LibraryContent.tsx
    │   └── router.ts
    ├── sidepanel/           # Side panel UI
    │   ├── App.tsx
    │   ├── LibraryView.tsx
    │   ├── GuideEditor.tsx
    │   ├── RecordingView.tsx
    │   ├── StepCard.tsx
    │   ├── ExportMenu.tsx
    │   ├── BlurCanvas.tsx
    │   └── ZoomScreenshot.tsx
    ├── onboarding/          # Onboarding wizard UI
    │   └── App.tsx          # 5-step wizard (welcome, AI, blur, pin, done)
    ├── shared/              # Shared UI components
    │   └── SettingsView.tsx  # AI settings (provider, model, API key)
    └── options/             # Settings page
        └── App.tsx

State Management

Layer Tool Purpose
Capture lifecycle xstate State machine (IDLE ↔ RECORDING) in background service worker
Fullview UI Zustand Search modal, guide counts, active guide data
Persistence Dexie (IndexedDB) Guides, steps, screenshots, snapshots
Service worker recovery sessionStorage xstate machine snapshot persistence
Background → Sidepanel Port messaging Real-time state broadcast
Cross-context sync BroadcastChannel Guide mutations (star, delete) across sidepanel/fullview

Extension Entry Points

Entry Point File Purpose
Background entrypoints/background/ Service worker: xstate actor, message handlers, tab management
Content Script entrypoints/content.ts Injected into all tabs: CaptureSession, event listeners
Side Panel entrypoints/sidepanel/ Recording controls, library, guide editor, settings
Full View entrypoints/fullview/ Dashboard: library browse, guide viewer, Ctrl+K search
Onboarding entrypoints/onboarding/ First-install wizard: AI setup, smart blur, pin extension
Options entrypoints/options/ Settings page (shared SettingsView in centered card)

Messaging

Content Script ←→ Background Service Worker ←→ Sidepanel / Fullview

Extension messages (webext-core, lib/messaging.ts):

  • getState → current capture state, step count, guide ID
  • startRecording({url}) → creates guide, returns guideId
  • stopRecording() → finalizes guide, generates AI title
  • captureStep({guideId, action, elementMeta, domContext}) → screenshots + creates step
  • updateInputStep({stepId, description}) → updates typing step description
  • finalizeInputStep({stepId, elementMeta, domContext}) → final screenshot + AI description

Tab messages (content script ↔ background, lib/tab-messages.ts):

  • PING / START_CAPTURE / STOP_CAPTURE — lifecycle
  • SHOW_NOTIFICATION — "Recording started" overlay
  • URL_CHANGED / GET_ROUTE — SPA navigation tracking

Capture Pipeline

Start recording:

  1. User clicks "Start Capture" in sidepanel
  2. Background transitions xstate machine IDLE → RECORDING
  3. Creates Guide in IndexedDB, broadcasts START_CAPTURE to all tabs
  4. Content scripts create CaptureSession → CaptureController (event listeners)
  5. Shows recording notification overlay on active tab

Capture a click:

  1. Content script's CaptureController detects click via DOM event listener
  2. Click handler pushes async work into PQueue (concurrency: 1)
  3. Enqueueing hides the hover ring (pointerdown already did, on mouse paths); queue waits 3 frames then sends captureStep
  4. Background calls captureVisibleTab (ring is hidden, page hasn't reacted yet)
  5. Saves Screenshot + Step to IndexedDB
  6. Queues the AI description from DOM context text (serialized in the background; the step carries aiPending until it lands) rather than awaiting it
  7. Returns { stepId } → queue processes next event; the ring returns only once the queue is fully drained

Capture text input (typing):

  1. Click on text field → CaptureController starts InputSession → captureStep with initial screenshot
  2. Each keystroke → InputSession.update() → updateInputStep (description only, fire-and-forget)
  3. Enter/Escape/focusout → InputSession.finalize() → finalizeInputStep → final screenshot replaces initial + queued AI description
  4. Result: one step for entire typing interaction

Stop recording:

  1. Background transitions RECORDING → IDLE
  2. Broadcasts STOP_CAPTURE, content scripts flush pending input sessions
  3. Background drains queued step descriptions (20s cap), then generates the guide title from step descriptions + URLs via AI
  4. Opens fullview dashboard with the guide

DOM Context (AI Input)

Instead of sending screenshots to the AI for step descriptions, Mimik extracts a lightweight DOM context (~50-100 tokens) from around the target element:

Page: "Public profile - Settings" /settings/profile
Container: form "Public profile"
Heading: "Public profile"
Siblings: input "Name", input "Email", textarea "Bio", button "Update profile"
→ Target: button "Update profile" (click)

Extraction walks up from the target element to find:

  • Page title + URL path
  • Nearest semantic container (form, nav, dialog, section) or 3 levels up
  • Nearest heading
  • Sibling interactive elements in the same container (max 10)

Export Formats

Format Generator Details
HTML core/export/html-export.ts Self-contained, base64 images, inline CSS
PDF core/export/pdf-export.ts jsPDF, A4 portrait, auto page breaks
Markdown core/export/markdown-export.ts Standard MD with base64 image data URLs
DOCX core/export/docx-export.ts Lazy-imported, Word-compatible
Video core/export/video-export.ts WebCodecs via mediabunny (lazy), mp4/H.264 with WebM/VP9 fallback

Video frames reuse renderScreenshot, so the auto-crop, click-target outline, annotations and redactions are already baked in. Each step holds 1.5s wide, eases into a crop around the target over 0.73s, holds 3s close, and consecutive steps cross-dissolve over 0.33s at 30fps. Capability probing lives in video-support.ts, which must stay free of mediabunny imports because both export UIs load it eagerly.

Tech Stack

Layer Technology
Extension framework WXT (Manifest V3)
Language TypeScript
UI React 19
Styling Tailwind CSS v4
Components shadcn/ui
State (capture) xstate
State (UI) Zustand
Storage Dexie.js (IndexedDB)
Messaging webext-core
Export jsPDF, docx, mediabunny (WebCodecs video), client-side HTML/Markdown
AI (optional) Vercel AI SDK (ai, @ai-sdk/openai, @ai-sdk/anthropic)
Event queue p-queue (concurrency: 1)
Icons Lucide React
Dates dayjs
DOM utils css-selector-generator

Design System

All colors are defined as CSS variables in src/ui/global.css and used via Tailwind classes:

Token Color Usage
--color-foreground #1E1B4B Primary text (deep navy)
--color-muted-foreground #6B7280 Secondary text
--color-border #C7D2FE Borders, dividers (lavender)
--color-secondary #EEF2FF Light wash backgrounds
--color-accent #4F46E5 Icons, links, focus rings, toggles, meters (indigo) — never a button fill
--color-primary #1E1B4B Primary buttons, badges, dark backgrounds
--color-primary-foreground #C7D2FE Text on dark backgrounds
--color-lavender #C7D2FE Soft accent
--color-purple #4F46E5 Primary indigo
--color-deep #1E1B4B Deepest navy
--color-violet #38BDF8 Sky blue accent
--color-success #059669 Success green

Font: Poppins (loaded via @fontsource/poppins).

Key Technical Details

  • Async event queue in content script (PQueue, concurrency 1) serializes capture work — each action awaits the background round-trip (screenshot + step write, not the AI description) before the next starts
  • Hover ring hidden when work is enqueued (CaptureController.enqueue, instant display:none), and show() stays suppressed until the queue drains — a second click while the first capture is still in flight can't bring the ring back before the screenshot. pointerdown hides it earlier still, on top of captureAction's 3-frame wait
  • Input session aggregates all typing on a field into one step — click creates it, keystrokes update description, finalize takes final screenshot
  • DOM context sent as text to AI instead of screenshots — 15-30x cheaper per step
  • Hover ring (lib/hover-ring.ts) is a closed-Shadow-DOM host marked data-mimik-ignore, shared by recording and the blur picker (purple). Recording reads the user's targetColor so the live ring matches the dashed target baked into screenshots. Never drawn on iframe/embed/object — a capture inside a subframe can't hide the top frame's ring
  • Content script injection pings first, falls back to chrome.scripting.executeScript() for tabs without the script
  • xstate snapshot persisted to sessionStorage so the state machine survives service worker restarts
  • Recording notification uses animationend event (not hardcoded delays) for timing
  • Font loading uses @fontsource/poppins (CSP-safe, no CDN dependency)
  • Cross-context sync via BroadcastChannel — star/delete events update other views without full reload
0