🤖 AI Note Taker

A modern macOS application that combines real-time speech transcription with AI-powered summaries. Built with React (TypeScript) frontend and Python (FastAPI) backend, powered by Deepgram for transcription and Google Gemini for AI insights.

✨ Features

🎤 Real-time Transcription - Live speech-to-text using Deepgram's advanced API
🔊 Speaker Audio Capture - Transcribe YouTube videos, music, or any system audio using BlackHole
🤖 AI-Powered Summaries - Intelligent meeting summaries using Google Gemini Flash 2.5
📝 Smart Insights - Extract action items, key points, and decisions automatically
🎨 Modern UI - Beautiful, responsive design optimized for macOS
🔒 Privacy-Focused - No bots in meetings, audio processed securely
⚡ Real-time Updates - See transcription as you speak with WebSocket connection
🎵 Multi-Output Audio - Hear audio while transcribing simultaneously
📋 Easy Export - Copy transcriptions and summaries to clipboard

🛠 Tech Stack

Frontend:

React 18 with TypeScript
Modern CSS with glassmorphism design
Lucide React icons
WebSocket for real-time communication

Backend:

Python FastAPI
Deepgram SDK for speech recognition
Google Generative AI (Gemini)
WebSocket support
Pydantic for data validation

📋 Prerequisites

Before you begin, ensure you have:

Python 3.8+ - Download Python
Node.js 16+ - Download Node.js
Deepgram API Key - Get free API key
Google Gemini API Key - Get API key
BlackHole (Optional) - For transcribing speaker audio (videos, music, etc.) instead of just microphone input

🚀 Quick Start

1. Clone and Setup

git clone <your-repo-url>
cd ai-note-taker

2. Backend Setup (Python)

# Install Python dependencies
pip install -r requirements.txt

# Create environment file
cp env.example .env

# Edit .env file with your API keys
# DEEPGRAM_API_KEY=your_deepgram_api_key_here
# GEMINI_API_KEY=your_gemini_api_key_here

3. Frontend Setup (React)

# Install Node.js dependencies
npm install

4. Get Your API Keys

Deepgram API Key:

Go to Deepgram Console
Sign up for a free account (includes $200 credit)
Create a new project
Copy your API key

Google Gemini API Key:

Go to Google AI Studio
Sign in with your Google account
Create a new API key
Copy your API key

5. Run the Application

Option A: Using the Startup Script (Recommended)

# Terminal 1 - Start Python Backend
./start_backend.sh

# Terminal 2 - Start React Frontend  
npm start

Option B: Manual Startup

# Terminal 1 - Start Python Backend
cd backend
source venv/bin/activate
uvicorn main:app --host 0.0.0.0 --port 8000

# Terminal 2 - Start React Frontend
npm start

Expected Output:

Backend runs on http://localhost:8000
Frontend runs on http://localhost:3000
You should see "INFO: Uvicorn running on http://0.0.0.0:8000" for backend
You should see "webpack compiled successfully" for frontend

6. Verify Everything is Working

Quick Status Check:

# Check if backend is running
curl http://localhost:8000/api/health

# Should return: {"status":"healthy","deepgram_configured":true,"gemini_configured":true}

Visual Confirmation:

Frontend: Open http://localhost:3000 - should show the AI Note Taker interface
Backend: Open http://localhost:8000 - should show API information
Both terminals should show no error messages

7. Use the App

Open http://localhost:3000 in your browser
Click "Start Recording" and allow microphone access
Start speaking - see real-time transcription
Click "AI Summary" to get intelligent insights
Use quick actions for specific extractions
Copy results or clear to start over

🔊 Speaker Audio Transcription (BlackHole Setup)

Want to transcribe audio from YouTube videos, music, or any speaker output? Follow this advanced setup to capture system audio instead of just microphone input.

Prerequisites for Speaker Audio

macOS (BlackHole is macOS-specific)
Homebrew - Install Homebrew if you don't have it

Step 1: Install BlackHole

# Install BlackHole virtual audio driver
brew install blackhole-2ch

# Note: You'll need to restart your Mac after installation
sudo reboot

Step 2: Create Multi-Output Device (Optional - for hearing + transcribing)

If you want to both hear audio AND transcribe it simultaneously:

Open Audio MIDI Setup:
```
open -a "Audio MIDI Setup"
```
Create Multi-Output Device:
- Click the "+" button → "Create Multi-Output Device"
- Check both boxes:
  - ✅ Your speakers (e.g., "Mac mini Speakers")
  - ✅ BlackHole 2ch
- Rename it to "Speakers + BlackHole"

Step 3: Configure Audio Routing

Option A: Transcribe Only (No Audio Playback)

# Set both input and output to BlackHole
# Output: All system audio goes to BlackHole (no speakers)
# Input: App captures audio from BlackHole

Option B: Hear + Transcribe Simultaneously

# Set output to Multi-Output Device (speakers + BlackHole)
# Set input to BlackHole 2ch
# Result: You hear audio AND app captures it

Step 4: Quick Audio Setup Commands

Install SwitchAudioSource for easy switching:

brew install switchaudio-osx

Quick Commands:

# Check available audio devices
SwitchAudioSource -a

# Set output to BlackHole only (transcribe only, no audio)
SwitchAudioSource -s "BlackHole 2ch"
SwitchAudioSource -t input -s "BlackHole 2ch"

# Set output to Multi-Output (hear + transcribe)
SwitchAudioSource -s "Speakers + BlackHole"
SwitchAudioSource -t input -s "BlackHole 2ch"

# Return to normal (speakers + microphone)
SwitchAudioSource -s "Mac mini Speakers"
SwitchAudioSource -t input -s "External Microphone"

Step 5: Test Speaker Audio Transcription

Configure audio routing (using commands above)
Play a YouTube video with clear speech
Open your transcription app: http://localhost:3000
Click "Start Recording"
Watch the video's audio get transcribed in real-time! 🎵→📝

How It Works

YouTube Video → System Audio → BlackHole → Your App → Deepgram → Transcription
     🎥              🔊           🔄         📱          🤖         📝

BlackHole acts as a "virtual cable" that routes system audio to your transcription app, enabling you to transcribe any audio playing on your computer.

🏗 Architecture

┌─────────────────┐    WebSocket    ┌─────────────────┐    API Calls    ┌─────────────────┐
│   React App     │ ◄──────────────► │  Python Backend │ ◄──────────────► │  Deepgram API   │
│   (Frontend)    │                 │   (FastAPI)     │                 │   (Speech AI)   │
│                 │                 │                 │                 └─────────────────┘
│                 │    HTTP/REST    │                 │    API Calls    ┌─────────────────┐
│                 │ ◄──────────────► │                 │ ◄──────────────► │  Gemini API     │
└─────────────────┘                 └─────────────────┘                 │   (AI Summary)  │
                                                                        └─────────────────┘

📁 Project Structure

ai-note-taker/
├── backend/
│   └── main.py              # Python FastAPI server
├── src/
│   ├── App.tsx              # Main React component
│   ├── App.css              # Styles for the app
│   ├── types.ts             # TypeScript type definitions
│   ├── index.tsx            # React app entry point
│   └── index.css            # Global styles
├── public/
│   └── index.html           # HTML template
├── package.json             # Node.js dependencies
├── requirements.txt         # Python dependencies
├── tsconfig.json           # TypeScript configuration
├── env.example             # Environment variables template
└── README.md               # This file

🔧 TypeScript Concepts Explained

Since you're new to TypeScript, here are the key concepts used in this project:

1. Interfaces - Data Structure Contracts

interface TranscriptionMessage {
  type: string;           // Required field
  text?: string;          // Optional field (note the ?)
  is_final?: boolean;     // Optional boolean
}

What this means: Interfaces define the shape of objects. The ? makes properties optional.

2. State with Types - Typed React State

const [isRecording, setIsRecording] = useState<boolean>(false);
const [transcription, setTranscription] = useState<string>('');

What this means: We tell TypeScript exactly what type of data each state variable holds.

3. Union Types - Multiple Possible Types

type ConnectionStatus = 'Disconnected' | 'Connected' | 'Connection Error';

What this means: A variable can be one of several specific string values.

4. Generic Types - Flexible Type Parameters

const websocketRef = useRef<WebSocket | null>(null);

What this means: The ref can hold either a WebSocket object OR null.

5. Function Types - Typed Functions

const connectWebSocket = (): Promise<void> => {
  // Returns a Promise that resolves to nothing (void)
};

🎯 API Endpoints

WebSocket Endpoints

ws://localhost:8000/ws - Real-time transcription

REST Endpoints

GET / - API information
GET /api/health - Health check
POST /api/summarize - Generate AI summary

Example API Usage

// Generate AI summary
const response = await fetch('http://localhost:8000/api/summarize', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    text: "Meeting transcript here...",
    summary_type: "meeting" // or "action_items" or "key_points"
  })
});

🔍 How It Works

Real-time Transcription Flow

Audio Capture: React captures microphone audio using MediaRecorder
WebSocket Streaming: Audio sent to Python backend via WebSocket
Deepgram Processing: Backend forwards audio to Deepgram's real-time API
Live Results: Deepgram returns both interim and final transcriptions
UI Updates: React displays results in real-time

AI Summary Generation

Text Processing: Complete transcript sent to backend
Gemini Analysis: Google's Gemini AI analyzes the content
Structured Output: AI returns formatted summaries, action items, etc.
Display: Frontend shows organized insights

🎨 UI Features

Glassmorphism Design - Modern translucent panels with blur effects
Real-time Status - Live connection status with animated indicators
Responsive Layout - Works on desktop and mobile devices
Copy Functionality - Easy clipboard integration
Loading States - Visual feedback during AI processing
Error Handling - Clear error messages and recovery options

🔧 Troubleshooting

Common Issues

1. Backend Server Won't Start or Stops Immediately

Symptoms: Backend starts but immediately exits, or shows "venv/bin/activate: No such file or directory"

Solutions:

# Make sure you're in the right directory
cd /path/to/your/RealtimeVoiceToText

# Use the startup script
chmod +x start_backend.sh
./start_backend.sh

# OR manually activate virtual environment
cd backend
source venv/bin/activate
uvicorn main:app --host 0.0.0.0 --port 8000

2. "Cannot find module 'react'" Error

npm install

3. "API Key not found" Error

Ensure .env file exists in backend/ directory (not root)
Add your actual API keys to backend/.env
Restart the Python backend

4. Microphone Permission Denied

Check browser microphone permissions
On macOS: System Preferences → Security & Privacy → Privacy → Microphone
Ensure browser has microphone access

5. WebSocket Connection Failed

Ensure Python backend is running on port 8000
Check that you see "INFO: Uvicorn running on http://0.0.0.0:8000"
Try restarting both frontend and backend
Check firewall settings

6. AI Summary Not Working

Verify Gemini API key is correct in backend/.env
Check backend logs for errors
Ensure you have API quota remaining

7. "It was working before" - Quick Fix

Backend probably stopped running
Use ./start_backend.sh to restart it
Frontend usually stays running and is fine

BlackHole & Speaker Audio Issues

6. No Transcription from Speaker Audio

Verify BlackHole is installed: SwitchAudioSource -a | grep BlackHole
Check audio routing: Output and Input both use BlackHole
Restart browser after changing audio settings
Check browser console (F12) for audio device selection

7. Can't Hear Audio While Transcribing

Use Multi-Output Device instead of BlackHole only
Ensure both speakers and BlackHole are checked in Multi-Output Device
Audio MIDI Setup → Multi-Output Device → Check both boxes

8. Browser Not Using BlackHole

Clear browser microphone permissions
Refresh page and manually select "BlackHole 2ch" when prompted
Chrome: Settings → Privacy → Microphone → Allow localhost:3000

9. BlackHole Not Appearing in Browser

Restart browser after installing BlackHole
Restart Mac if BlackHole still not visible
Check System Preferences → Sound → Input for BlackHole

10. Audio Cutting Out or Poor Quality

Reduce audio constraints in browser (disable echo cancellation for system audio)
Check CPU usage - transcription is resource-intensive
Try lower sample rate in MediaRecorder settings

Development Commands

# Backend development with auto-reload
cd backend && uvicorn main:app --reload

# Frontend development
npm start

# Install new Python package
pip install package_name && pip freeze > requirements.txt

# Install new Node package
npm install package_name

🚀 Deployment

For macOS App Distribution

Build React App

npm run build

Package Python Backend

pip install pyinstaller
pyinstaller --onefile backend/main.py

Create macOS App Bundle

Use tools like py2app or electron to create native macOS app
Include both frontend build and backend executable

🤝 Contributing

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

Deepgram for excellent speech-to-text API
Google AI for Gemini AI capabilities
FastAPI for the modern Python web framework
React for the powerful frontend library
Lucide for beautiful icons

📞 Support

If you encounter issues:

Check the troubleshooting section above
Look at browser console for error messages
Check Python backend logs
Create an issue in this repository

Happy note-taking! 🎤→📝→🤖

Name		Name	Last commit message	Last commit date
.vscode		.vscode
assets		assets
backend		backend
public		public
src		src
.gitignore		.gitignore
DEEPGRAM_ARCHITECTURE_GUIDE.md		DEEPGRAM_ARCHITECTURE_GUIDE.md
DEPLOYMENT_PLATFORMS.md		DEPLOYMENT_PLATFORMS.md
LICENSE		LICENSE
MACOS_APP_README.md		MACOS_APP_README.md
PACKAGE_EXPLAINED.md		PACKAGE_EXPLAINED.md
QUICKSTART.md		QUICKSTART.md
README.md		README.md
env.example		env.example
package-lock.json		package-lock.json
package.json		package.json
requirements.txt		requirements.txt
setup.sh		setup.sh
start_backend.sh		start_backend.sh
strictmode-comparison.jsx		strictmode-comparison.jsx
strictmode-demo.jsx		strictmode-demo.jsx
tsconfig.json		tsconfig.json

Latest commit

History

Folders and files

Repository files navigation