Upload a video of a weightlifting set, paste the exercise you programmed, and get back a rep count and coaching notes from Gemini.
Note: This repository is almost entirely AI-coded.
You give it a video and a description of the exercise ("Snatch Complex — 6 sets. Each rep: one slow snatch high pull, one squat snatch, one snatch balance"). It returns:
- A rep count — how many complete reps of the programmed exercise it saw.
- A rep breakdown — every rep attempted, with each movement timestamped and marked complete, partial, or missed. Click a timestamp to jump the player there.
- Form feedback — coaching notes grouped under the specific movement they concern, tagged as something you're doing well, a refinement, or a fault to fix first.
The model is told what one rep consists of, so it handles both compound movements (a complex chaining several lifts) and single movements (a standalone squat), rendering each appropriately.
Requires Node 20+ and ffmpeg on your PATH (used to
convert .mov uploads — iPhone videos — into .mp4 before upload).
npm installCreate a .env in the repo root with a
Gemini API key:
GEMINI_API_KEY=your-key-here
Then:
npm run devOpen http://localhost:3000, drag in a video, and describe the exercise.
Each analysis costs roughly $0.02 in Gemini API usage. The exact token count and estimated cost are shown under the results.
Inference runs server-side so the API key never reaches the browser.
| Path | Role |
|---|---|
app/page.tsx |
Client UI — upload state, video player, layout |
app/actions.ts |
Server action: saves the upload, runs the analysis |
lib/gemini.ts |
Gemini upload/poll/analyze, response schema, cost math |
lib/video.ts |
.mov → .mp4 conversion via ffmpeg |
components/ |
Upload form, rep breakdown, feedback groups |
The uploaded video is written to uploads/ on the host and kept; the copy sent
to Gemini is deleted from their Files API once the analysis returns. uploads/
and all video files are gitignored.
Gemini is constrained by a JSON schema rather than asked for prose. The model first declares the movements making up one rep, then references that list by index from both the reps and the feedback — so a note can never be attached to a movement the model didn't identify. Responses are validated on the way in: unresolvable references are dropped rather than rendered wrong.
scripts/test_workout_video.py is the original
command-line harness that predates the app. It reads exercises from
scripts/workouts.toml and prints the model's answer
plus token cost. Still useful for iterating on prompts without the UI.
Needs Python 3.11+ (for tomllib) — note that macOS system python3 is 3.9.
python3.12 -m pip install -r scripts/requirements.txt
python3.12 scripts/test_workout_video.pyVideo paths in workouts.toml resolve from the repo root.
The script has a --skeletal-overlay flag that renders MediaPipe pose landmarks
onto the video before sending it to Gemini. This was tried and abandoned — it
did not improve the analysis, so the web app sends raw video only. The flag
remains in the script for anyone who wants to re-test the idea.
- Rep counts vary between runs. The same video analyzed twice has returned different counts, because the model's judgment on whether a movement was "completed to standard" shifts. Treat the count as a strong hint, not a measurement.
- No persistence — results live only in the page until you upload another video.
- No auth, no rate limiting. Local tool, not deployed as-is.