A local-first meeting notetaker in the spirit of Granola: it captures your meeting audio on-device, transcribes it locally, lets you type your own rough notes as you go, and then merges the transcript with your notes into one polished, structured summary.
Two things are deliberate and worth stating up front:
- No bot joins your call. On Linux, capture taps the monitor of your default audio sink directly through PipeWire. Nothing is added as a meeting participant.
- It augments your notes rather than replacing them. You keep typing shorthand; the model's job is to merge, not to invent. The prompt explicitly forbids inventing decisions, owners, or action items.
The whole product is one Rust process:
| Concern | Choice |
|---|---|
| GUI | Iced (wgpu renderer) |
| Audio capture | pipewire-rs |
| Transcription | whisper-rs (whisper.cpp) |
| Storage | SQLite via rusqlite |
| Summaries | OpenAI or Anthropic, behind a Summarizer trait |
The UI, audio, and transcription code share one address space and exchange plain Rust values over channels. There is no IPC boundary, no serialization layer, no Node.js, and no Python sidecar to supervise.
┌─ Iced application (one process) ─────────────────────────────────┐
│ UI thread Message::update() -> view() │
│ (meeting list, notes editor, transcript, summary)│
│ Capture worker pipewire-rs: default sink monitor + microphone │
│ -> mix/resample to 16 kHz mono │
│ STT worker whisper-rs on a dedicated thread │
│ -> VAD-gated chunker -> partial + final segments │
│ LLM worker reqwest, only on demand │
│ Storage rusqlite │
└──────────────────────────────────────────────────────────────────┘
workers -> std::sync::mpsc -> 100 ms tick -> update()
Capture runs on PipeWire's own realtime threads. The frame callback does only two cheap
things: report a level, and hand the frame to a bounded channel. Everything expensive lives
on a dedicated notetaker-transcribe thread — mixing, resampling, archiving to WAV, and
Whisper inference. Inference is therefore not on the UI thread, and not on the audio
thread either, so a slow decode can never stall the meters or drop audio.
Meters are driven straight from the frame callback for that reason: they stay smooth while a decode is in flight. The recording timer is wall-clock time, not audio time, so it cannot drift when the decoder falls behind.
The microphone and the system output are captured as separate streams. That gives a cheap speaker attribution without a diarisation model: a segment is labelled You when the mic was dominant and Them when the system output was. Overlapping speech falls back to Speaker. Full diarisation is out of scope.
- Every frame is buffered. The cheap energy VAD only decides where utterances are cut, never what is kept — gating the buffer on it is what used to drop quiet speech.
- While speech continues, the accumulated audio is decoded periodically into a partial line, shown greyed out and then replaced.
- When a run of silence follows, or the utterance grows past a ceiling, the buffer is trimmed to the last voiced sample and decoded once more as a final line, which is what gets persisted. Chunks containing no voiced audio are discarded rather than sent to the decoder, since decoding pure silence only invites hallucinations.
The level recorded per frame is accumulated over the utterance, and that is what picks the speaker for the final.
- Rust 1.80 or newer.
- A C/C++ toolchain (
gcc/clang),pkg-config, andcmake— required bylibsqlite3-sys,whisper.cpp, and TLS crates. - Linux: PipeWire development headers.
libclangis only needed if you letwhisper-rs-sysgenerate its bindings; see below.
PipeWire version note.
libspa-sysgenerates its bindings from your installed SPA headers at build time, so the crate version must understand those headers. This project usespipewire0.10, which handles both the 0.3.x and 1.x layouts. Older revisions (0.8 and earlier) fail against PipeWire 1.x withno field `data` on type `spa_pod_builder`. If you need APIs newer than your default feature set, enable the matching feature, e.g.pipewire = { version = "0.10", features = ["v1_0_0"] }.
Debian/Ubuntu:
sudo apt install build-essential pkg-config cmake libpipewire-0.3-dev libclang-dev
Fedora:
sudo dnf install gcc-c++ make pkgconf-pkg-config cmake pipewire-devel clang-devel
sqlite3 itself is not required: the rusqlite bundled feature compiles SQLite
from source.
The native backends are behind feature flags, so the crate always builds even on a machine without PipeWire or Whisper toolchains:
# UI only: mock capture and mock transcription (no native deps beyond the GUI).
cargo run
# Real capture + local transcription.
cargo build --release --features pipewire,whisper
cargo run --release --features pipewire,whisper
whisper-rs-sys runs bindgen over its vendored whisper.h at build time, which
requires libclang and is sensitive to the installed toolchain. On some setups bindgen
silently emits an opaque whisper_full_params (a lone _address field) instead of
the real struct, and the build then fails with messages like:
error[E0609]: no field `n_threads` on type `whisper_full_params`
= note: available field is: `_address`
The project's .cargo/config.toml sets WHISPER_DONT_GENERATE_BINDINGS=1, which makes
the crate skip bindgen and use the bindings its author generated from the same vendored
sources. That fixes the build and drops the libclang requirement, and it keeps Whisper
on the CPU, which is what we want anyway so it does not contend with the UI's GPU
renderer.
Two things to know:
- Force one rebuild if
whisper-rs-syshas already run. The crate does not declare this variable viacargo:rerun-if-env-changed, so cargo will keep reusing a cached badbindings.rs. Runcargo clean -p whisper-rs-sysonce (orcargo clean). - Opting out. The crate tests for the variable's presence, so setting it to
0would not disable it — comment the line out instead. You need to do this if you enablewhisper-rs/vulkan, because the pre-generated bindings contain no Vulkan entry points.
| Feature | Default | Effect when off |
|---|---|---|
ui |
yes | The binary is not built; the library still compiles. |
pipewire |
no | Capture falls back to the mock backend; capture_is_available() is false. |
whisper |
no | Transcription falls back to the mock engine. |
With everything off you get a fully working application that uses synthetic audio and placeholder transcripts. That is intentional: the pipeline, database, and UI stay testable on any machine.
cargo test --lib # core: models, db, audio, chunker, LLM, prompts
cargo test --lib --all-features # additionally exercises the real backends
- Launch the app and press New meeting.
- Press Record and talk. The transcript streams in; type your own notes alongside.
- Press Stop. The mixed audio is archived and the transcript is committed.
- Press Summarise. The transcript and your notes are sent to the configured model.
- Export writes the summary and your notes to Markdown under the data directory.
Open Settings and pick a provider. Keys can be typed in, but the environment is always preferred and keeps secrets out of the config file:
export OPENAI_API_KEY=...
export ANTHROPIC_API_KEY=...
export OPENROUTER_API_KEY=...
| Provider | Models are named | Notes |
|---|---|---|
| OpenAI | gpt-4o, gpt-4o-mini |
Sends response_format: json_object. |
| Anthropic | claude-sonnet-4-5, … |
Uses the Messages API. |
| OpenRouter | openai/gpt-4o, anthropic/claude-…, … |
One key for many vendors' models. Model ids are namespaced vendor/model. |
| Mock (offline) | — | No key, no network. |
OpenRouter does not send response_format: json_object, because only some upstream
models honour it and others reject the request outright. It relies on the prompt and the
tolerant response parser instead, so almost any model listed on OpenRouter will work —
including the cheap ones, which is handy for the live pass.
The Mock (offline) provider needs no key and performs real (if simple) extraction:
- [ ] checkboxes and TODO:/action: lines become action items, and decided: lines
become decisions. This keeps the app useful and testable without a network.
Whisper models are downloaded on demand into the data directory, not bundled. The
default is large-v3-turbo (near-large-v3 quality, much lower latency). For a first
run, base or small downloads far faster. Choose one in Settings and press
Download model. Downloads go to a .part file and are renamed only on success.
Accuracy, for reference:
| Model | Size | Notes |
|---|---|---|
tiny / base |
75–142 MB | Fine for a smoke test; noticeably lossy on real meetings. |
small |
466 MB | A clear step up, still quick. A good everyday choice. |
medium |
1.5 GB | Better again, slower on CPU. |
large-v3-turbo |
1.6 GB | Default. Large-v3 quality at much lower latency. |
large-v3 |
3.1 GB | Best accuracy, heaviest. |
Two practical notes:
- Changing the model requires pressing Save settings, then Download model — the running decoder is built from saved config when you press Record.
- Accuracy is limited by more than the model: the energy VAD splits utterances at
stt.silence_end_ms(default 700 ms), and each partial decode re-reads the growing buffer. The segmenter is tunable inconfig.json— raisestt.partial_interval_ms(default 1200) if the same text keeps being re-decoded, raisestt.max_chunk_ms(default 8000) so long monologues run longer before a forced final, lowerstt.silence_db(default −70) if quiet speech is treated as silence, and adjuststt.min_speech_ms(default 300) to change how short a blip must be to count as noise. These are read when recording starts.
When built without --features whisper, the mock engine is used and no download is
needed.
Everything stays on the machine, under the platform data directory
(~/.local/share/notetaker on Linux):
notetaker.db meetings, transcripts, notes, summaries
audio/ one WAV per meeting
models/ downloaded Whisper models
exports/ Markdown exports
Config lives at ~/.config/notetaker/config.json.
Iced renders the UI with wgpu, and Whisper can also use a GPU. The per-decoder Use GPU setting is off by default so Whisper stays on the CPU and the two do not compete.
There is no runtime renderer switch: in Iced 0.13 the renderer is selected at compile time by feature flags, and the default build uses wgpu with an automatic software fallback. To force software rendering and leave the GPU entirely free, build with wgpu disabled:
# Cargo.toml
iced = { version = "0.13", default-features = false, features = ["tiny-skia", "tokio", "image"] }Run with logging on, then press Record and speak for a few seconds:
RUST_LOG=notetaker=debug cargo run --release --features pipewire,whisper
The messages appear in order, and whichever one is missing tells you where it stops:
| Expected log line | Meaning if absent |
|---|---|
starting capture via PipeWire … |
Recording never started — check for an error banner in the UI. |
transcription worker ready (…) |
The worker thread failed to start (the message says which engine). |
system audio linked: … Hz, … channel(s), format … |
The stream never linked to the sink's monitor. |
microphone linked: … |
The stream never linked to the default source. |
… first frame received (N samples, …) |
Linked, but no audio ever arrived. |
transcription is falling behind; N audio frames dropped |
Inference is slower than real time; use a smaller model or raise partial_interval_ms. |
audio level X dB (silence threshold Y dB), speech=… |
Emitted about once a second. If X never rises above Y, the VAD never opens and nothing is transcribed — lower stt.silence_db in the config fil
5B9C
e. |
Two useful cross-checks while it runs:
- Do the level meters move? They are driven directly by captured audio, so live meters with no transcript point at transcription, while dead meters point at capture.
- Is the archive growing?
ls -l ~/.local/share/notetaker/audio/shows the WAV being written; its size should increase by roughly 32 KB per second while recording.
If the log shows a negotiated format other than F32LE, capture will refuse to
interpret it and say so — that means the graph ignored our format request and the
sample reader needs extending.
- Speaker attribution is two-way, not true diarisation. Overlapping speech is
labelled
Speaker. - VAD is energy-based. whisper.cpp's Silero VAD is the natural upgrade and is what
the
ChunkerConfigis shaped around. - Live summaries are not wired to the UI yet.
LlmConfig::live_summaryand the two-tier model fields exist, andSummarizeris cheap enough to call on a timer, but only the end-of-meeting pass is currently triggered. - Capture is Linux/PipeWire only. macOS (ScreenCaptureKit) and Windows (WASAPI
loopback) backends belong behind the same
CaptureBackendtrait and are not written. - Packaging is unhandled. There is no installer, and as a Flatpak the app would need explicit PipeWire socket permissions to capture.
cargo fmt --check
cargo clippy --all-targets --features pipewire,whisper -- -D warnings
cargo test --lib --all-features