A macOS menu bar companion that sees your screen, knows your company's docs, and walks you through any issue β voice or text.
How it works: Employee presses a hotkey β app captures their screen + transcribes their voice β retrieves relevant company docs from the knowledge base β sends everything to GPT-4o β responds with voice and points at UI elements on screen.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β macOS App (Swift/SwiftUI) β
β Push-to-talk β Screenshot β AI Response β TTS β Pointing β
ββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββ
β all requests
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Cloudflare Worker (:8787) β
β Single gateway β app never calls external APIs directly β
β /chat β /tts β /transcribe β /query-knowledge β
ββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββ
β
ββββββββββββββΌβββββββββββββ
βΌ βΌ βΌ
βββββββββββ βββββββββββ ββββββββββββ
β OpenAI β β RAG β β MinIO β
β APIs β β Service β β Storage β
β β β (:8000) β β (:9000) β
βββββββββββ ββββββ¬βββββ ββββββββββββ
β
ββββββββΌβββββββ
βΌ βΌ βΌ
Langfuse Sentry OpenAI
Tracing Errors Vector
Stores
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Frontend (:4000) β Next.js β
β /create β create company + upload docs + setup token β
β /login β returning admin enters API key β
β /dashboard β usage, setup link, knowledge base stats β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
- Docker & Docker Compose
- Xcode 16+ (for the macOS app)
- An OpenAI API key
git clone https://github.com/harikesavan/Pookify.git
cd PookifyCreate the environment files:
# Root .env (used by docker-compose for the Worker)
cat > .env << 'EOF'
OPENAI_API_KEY=sk-proj-your-key-here
EOF
# RAG service .env
cat > rag-service/.env << 'EOF'
OPENAI_API_KEY=sk-proj-your-key-here
LANGFUSE_SECRET_KEY=sk-lf-your-key
LANGFUSE_PUBLIC_KEY=pk-lf-your-key
LANGFUSE_BASE_URL=https://cloud.langfuse.com
SENTRY_DSN=https://your-dsn@sentry.io/project
EOFdocker compose up --build -dThis starts 4 containers:
| Service | Port | Purpose |
|---|---|---|
| MinIO | :9000 / :9001 | S3-compatible file storage |
| RAG Service | :8000 | Knowledge base, company management, AI chat proxy |
| Worker | :8787 | Single gateway for the macOS app |
| Frontend | :4000 | Admin dashboard |
open leanring-buddy.xcodeproj
# Select leanring-buddy scheme β set signing team β Cmd+R- Open
http://localhost:4000/create - Enter company name + description
- Upload knowledge documents (PDF, DOCX, TXT, MD)
- Click "Build my Pookie"
- You'll be redirected to the dashboard
- Click "Open in Pookie" to configure the macOS app
Pookify/
βββ leanring-buddy/ macOS app (Swift/SwiftUI)
β βββ App/ Entry point, CompanionManager, CompanyConfig
β βββ Voice/ Push-to-talk, transcription providers
β βββ AI/ OpenAI API clients, KnowledgeBaseClient
β βββ UI/ Menu bar panel, overlay, design system
β βββ Utilities/ Screenshots, permissions, analytics
β βββ Resources/ Assets, audio files
β
βββ rag-service/ Python RAG service (FastAPI)
β βββ app/
β β βββ config.py Environment variables
β β βββ models.py Pydantic request models
β β βββ storage.py JSON persistence + MinIO client
β β βββ companies.py Company CRUD + API key management
β β βββ ingestion.py File upload β OpenAI vector store
β β βββ search.py Vector store search (Langfuse traced)
β β βββ chat.py Chat completions proxy (Langfuse traced)
β β βββ setup_tokens.py One-time setup token generation
β β βββ routes.py FastAPI route handlers
β β βββ observability.py Langfuse + Sentry initialization
β βββ main.py FastAPI entry point
β βββ Dockerfile
β βββ requirements.txt
β
βββ worker/ Cloudflare Worker (TypeScript)
β βββ src/index.ts Single gateway proxy
β βββ Dockerfile
β βββ wrangler.toml
β
βββ frontend/ Admin dashboard (Next.js)
β βββ src/app/
β β βββ page.tsx Landing page
β β βββ create/ Company creation flow
β β βββ (auth)/login/ API key login
β β βββ (auth)/dashboard/ Company dashboard
β β βββ api/ Backend API routes
β βββ Dockerfile
β
βββ docker-compose.yml All 4 services + MinIO + volumes
1. User presses Ctrl+Option (push-to-talk)
2. AVAudioEngine captures microphone audio
3. Audio uploaded to OpenAI transcription β text transcript
4. App captures all connected screens via ScreenCaptureKit
5. Transcript sent to RAG service (via Worker) β retrieves relevant company docs
6. GPT-4o receives: system prompt + retrieved docs + screenshots + transcript
7. AI responds with text + optional [POINT:x,y:label] tags
8. Text sent to OpenAI TTS β audio played back
9. If pointing tags present β blue cursor animates to the UI element
Each company gets:
- Its own OpenAI vector store (documents never mix)
- A unique API key (
ck_live_xxx) - Custom system prompt instructions
- Isolated usage tracking
New company (admin):
/create β fill form + upload docs β company created in RAG service
β API key saved to browser β redirected to /dashboard
Returning admin:
/login β enter API key β validated against RAG service β /dashboard
Employee setup:
Admin clicks "Open in Pookie" on dashboard
β generates one-time setup token (15min expiry)
β opens pookify://setup?token=pst_xxx
β macOS app exchanges token for real API key (token deleted)
β app is configured, no further setup needed
| Tool | What It Tracks |
|---|---|
| Langfuse | Every OpenAI call β prompts, responses, tokens, cost, latency. RAG retrieval spans. Session grouping. |
| Sentry | Errors, request traces, OpenAI call spans across the full stack. |
Dashboards:
- Langfuse: cloud.langfuse.com
- Sentry: Your project at sentry.io
Company Management (admin dashboard)
| Method | Endpoint | Purpose |
|---|---|---|
| POST | /companies | Create company + vector store + API key |
| GET | /companies | List all companies |
| GET | /companies/{id} | Get company details + file count |
| PATCH | /companies/{id} | Update name or custom instructions |
| POST | /companies/{id}/regenerate-key | Rotate API key |
| GET | /companies/{id}/config | Get app config JSON |
| POST | /companies/{id}/setup-token | Generate one-time setup token |
| POST | /companies/{id}/ingest | Upload file to vector store |
| /companies/{id}/ingest-from-minio | Ingest from MinIO |
Query (macOS app via Worker, requires x-api-key header)
| Method | Endpoint | Purpose |
|---|---|---|
| POST | /query | Search company's vector store, return chunks |
| POST | /ask | Full RAG: search + GPT-4o answer + citations |
| POST | /chat | Proxy chat completions (Langfuse traced) |
| POST | /exchange-token | Exchange setup token for API key (one-time) |
System
| Method | Endpoint | Purpose |
|---|---|---|
| GET | /health | Service status, company count, MinIO status |
All routes proxy to either OpenAI or the RAG service. The macOS app only talks to this.
| Route | Upstream |
|---|---|
| POST /chat | RAG Service /chat β OpenAI (Langfuse traced) |
| POST /tts | OpenAI TTS |
| POST /transcribe | OpenAI Audio Transcription |
| POST /query-knowledge | RAG Service /query |
| POST /exchange-token | RAG Service /exchange-token |
OPENAI_API_KEY=sk-proj-...
OPENAI_API_KEY=sk-proj-...
LANGFUSE_SECRET_KEY=sk-lf-... # optional
LANGFUSE_PUBLIC_KEY=pk-lf-... # optional
LANGFUSE_BASE_URL=https://cloud.langfuse.com
SENTRY_DSN=https://...@sentry.io/... # optional
MINIO_ROOT_USER=minioadmin
MINIO_ROOT_PASSWORD=minioadmin
MinIO console: http://localhost:9001
| Layer | Technology |
|---|---|
| macOS App | Swift, SwiftUI, AppKit, ScreenCaptureKit, AVAudioEngine |
| AI Model | OpenAI GPT-4o (vision + chat) |
| Speech-to-Text | OpenAI Audio Transcription |
| Text-to-Speech | OpenAI TTS |
| Knowledge Base | OpenAI Vector Stores + File Search |
| API Gateway | Cloudflare Worker (TypeScript) |
| RAG Backend | Python, FastAPI |
| Observability | Langfuse (LLM tracing), Sentry (error monitoring) |
| File Storage | MinIO (S3-compatible) |
| Frontend | Next.js 16, React 19, Tailwind CSS, shadcn/ui |
| Containers | Docker Compose |
# RAG service (without Docker)
cd rag-service
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
uvicorn main:app --port 8000
# Worker (without Docker)
cd worker
npm install
npx wrangler dev --port 8787
# Frontend (without Docker)
cd frontend
npm install
npm run dev- Do NOT run
xcodebuildfrom the terminal β invalidates TCC permissions - The app uses
PBXFileSystemSynchronizedRootGroupβ Xcode auto-syncs the directory structure - Push-to-talk shortcut: Ctrl+Option (hold to talk, release to send)
- The
pookify://URL scheme is registered in Info.plist for setup token exchange