Field Notes backend for voice-driven structured data capture. Turns your Omi wearable into a voice-driven reporting assistant for technicians, real-estate agents, contractors, and field engineers.
Instead of juggling clipboards, phones, or spreadsheets, just speak your observations, and Omi instantly structures and logs them into Google Sheets — ready for reports, invoices, or client updates.
- Voice-Driven Data Capture: Process transcript segments from Omi wearable devices
- AI-Powered Extraction: Use OpenAI GPT models to extract structured data from natural language
- Google Sheets Integration: Automatically append extracted data to Google Sheets
- User-Specific Storage: Isolated data storage per user with OAuth authentication
- Dynamic Schema Generation: LLM-generated Pydantic models based on user prompts
- RESTful API: FastAPI-based backend with comprehensive endpoints
- Background Processing: Asynchronous transcript processing for scalability
- CLI Tools: Command-line utilities for token management
- Standalone Demo: Test the application with a provided OpenAI key on the main page
- Added standalone demo on main page with OpenAI key provided by user.
- How to run standalone demo:
- Open with any uid: http://127.0.0.1:8000/oauth/start?uid=123
- Submit the form with the same uid: http://127.0.0.1:8000/
- How to run standalone demo:
https://h.omi.me/apps/01K8WEY210BJFDD7X2TZ70MVA1
- Python 3.13+
- uv package manager
- Google Cloud Console project with OAuth 2.0 credentials
- OpenAI API key
-
Clone the repository:
git clone <repository-url> cd omi-notes
-
Install dependencies:
uv sync
-
Configure environment variables:
cp .env.example .env
Edit
.envwith your credentials:# OpenAI API Configuration OPENAI_API_KEY=your_openai_api_key_here # Google OAuth Configuration GOOGLE_CLIENT_ID=your_google_client_id_here GOOGLE_CLIENT_SECRET=your_google_client_secret_here GOOGLE_REDIRECT_URI=http://localhost:8000/oauth/callback # OMI Key (if needed) OMI_KEY=your_omi_key_here
Development:
uv run uvicorn main:app --reloadProduction:
uv run uvicorn main:app --host 0.0.0.0 --port 8000- Setup Google OAuth: Visit
http://localhost:8000/oauth/start?uid=<user_id>to authenticate - Configure Extraction Prompt: Use the web interface or API to set your data extraction instructions
- Link Google Sheet: Provide a Google Sheets URL to store extracted data
- Send Transcripts: POST transcript segments to
/transcriptendpoint - View Results: Data automatically appears in your linked Google Sheet
GET /- Homepage with setup instructionsPOST /transcript- Receive and process transcript segmentsPOST /instructions- Store user extraction promptsGET /api/instructions- Retrieve user extraction promptsPOST /api/sheet- Store Google Sheet ID from URLGET /api/sheets- List user's Google Sheets
GET /oauth/start- Initiate Google OAuth flowGET /oauth/callback- Handle OAuth callback
GET /sheets- Google Sheets selection/creation pageGET /privacy- Privacy policyGET /terms- Terms of service
curl -X POST "http://localhost:8000/instructions" \
-H "Content-Type: application/json" \
-d '{
"uid": "user123",
"prompt": "Extract item name, quantity, and condition from the transcript."
}'curl -X POST "http://localhost:8000/transcript" \
-H "Content-Type: application/json" \
-d '{
"session_id": "user123",
"segments": [
{"id": "seg1", "text": "Found 5 damaged cables in the server room"},
{"id": "seg2", "text": "Location is building A, floor 3"}
],
"location": "Building A, Floor 3",
"datetime": "2024-01-15T10:30:00Z"
}'curl -X POST "http://localhost:8000/api/sheet" \
-H "Content-Type: application/json" \
-d '{
"session_id": "user123",
"sheet_url": "https://docs.google.com/spreadsheets/d/1ABC...XYZ/edit"
}'uv run python cli.py refresh-token user123| Variable | Description | Required |
|---|---|---|
OPENAI_API_KEY |
OpenAI API key for LLM processing | Yes |
GOOGLE_CLIENT_ID |
Google OAuth client ID | Yes |
GOOGLE_CLIENT_SECRET |
Google OAuth client secret | Yes |
GOOGLE_REDIRECT_URI |
OAuth callback URL | No (defaults to localhost:8000) |
OMI_KEY |
Omi device API key | Optional |
- Go to Google Cloud Console
- Create a new project or select existing one
- Enable Google Sheets API and Google Drive API
- Create OAuth 2.0 credentials
- Add authorized redirect URIs (include your callback URL)
- Set appropriate scopes in the application
- User data is stored in the
data/directory - Each user gets their own subdirectory:
data/<user_id>/ - Files stored per user:
prompt.txt- Extraction instructionstokens.json- OAuth credentialssheet_id.txt- Linked Google Sheet IDmodel_*.py- Cached Pydantic models
The project includes systemd service files for production deployment:
-
Copy service files:
sudo cp systemd/ominotes.service /etc/systemd/system/ sudo cp systemd/ominotes.ini /etc/systemd/system/
-
Create user and directories:
sudo useradd -r -s /bin/false ominotes sudo mkdir -p /home/ominotes/main sudo mkdir -p /var/log/ominotes sudo mkdir -p /run/ominotes sudo chown -R ominotes:ominotes /home/ominotes /var/log/ominotes /run/ominotes
-
Copy application and configure environment:
sudo cp -r . /home/ominotes/main/ sudo cp .env /home/ominotes/ sudo chown ominotes:ominotes /home/ominotes/.env -
Enable and start service:
sudo systemctl daemon-reload sudo systemctl enable ominotes sudo systemctl start ominotes
FROM python:3.13-slim
WORKDIR /app
COPY . .
RUN pip install uv && uv sync --no-dev
EXPOSE 8000
CMD ["uv", "run", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]- FastAPI Application (
main.py): Core web server and API endpoints - Data Processing: Background tasks for transcript processing
- LLM Integration: OpenAI GPT models for data extraction
- Google APIs: Sheets API for data storage, Drive API for sheet management
- User Management: Session-based user isolation with persistent storage
- CLI Tools (
cli.py): Administrative utilities
- Transcript Reception: API receives transcript segments with metadata
- User Context Loading: Retrieve user's prompt, OAuth tokens, and sheet configuration
- LLM Processing: Generate or load cached Pydantic model based on user prompt
- Data Extraction: Apply LLM chain to extract structured data from transcript
- Sheet Integration: Append extracted data to configured Google Sheet
- Token Management: Refresh OAuth tokens as needed
- User data isolation in separate directories
- OAuth 2.0 for Google API access
- Environment variable configuration
- No sensitive data logging
- Background processing for scalability
omi-notes/
├── main.py # FastAPI application
├── cli.py # CLI tools
├── pyproject.toml # Project configuration
├── uv.lock # Dependency lock file
├── .env.example # Environment template
├── data/ # User data storage
├── templates/ # Jinja2 templates
│ ├── index.html
│ ├── sheets.html
│ ├── privacy.html
│ ├── terms.html
│ └── default_prompt.txt
├── systemd/ # Systemd service files
└── README.md
- Define API endpoints in
main.py - Add Pydantic models for request/response data
- Implement business logic with proper error handling
- Update user storage functions if needed
- Add CLI commands in
cli.pyif required - Update documentation
This project is licensed under the MIT License - see the LICENSE file for details.
[Add contribution guidelines here]
[Add support information here]