8000 GitHub - gislik/notetaker · GitHub
[go: up one dir, main page]

Skip to content
gislikPublic

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

2 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Notetaker

A local-first meeting notetaker in the spirit of Granola: it captures your meeting audio on-device, transcribes it locally, lets you type your own rough notes as you go, and then merges the transcript with your notes into one polished, structured summary.

Two things are deliberate and worth stating up front:

  • No bot joins your call. On Linux, capture taps the monitor of your default audio sink directly through PipeWire. Nothing is added as a meeting participant.
  • It augments your notes rather than replacing them. You keep typing shorthand; the model's job is to merge, not to invent. The prompt explicitly forbids inventing decisions, owners, or action items.

Why there is no Python, and no webview

The whole product is one Rust process:

Concern Choice
GUI Iced (wgpu renderer)
Audio capture pipewire-rs
Transcription whisper-rs (whisper.cpp)
Storage SQLite via rusqlite
Summaries OpenAI or Anthropic, behind a Summarizer trait

The UI, audio, and transcription code share one address space and exchange plain Rust values over channels. There is no IPC boundary, no serialization layer, no Node.js, and no Python sidecar to supervise.

Architecture

┌─ Iced application (one process) ─────────────────────────────────┐
│  UI thread      Message::update() -> view()                      │
│                 (meeting list, notes editor, transcript, summary)│
│  Capture worker pipewire-rs: default sink monitor + microphone   │
│                 -> mix/resample to 16 kHz mono                   │
│  STT worker     whisper-rs on a dedicated thread                 │
│                 -> VAD-gated chunker -> partial + final segments │
│  LLM worker     reqwest, only on demand                          │
│  Storage        rusqlite                                         │
└──────────────────────────────────────────────────────────────────┘
        workers -> std::sync::mpsc -> 100 ms tick -> update()

Capture runs on PipeWire's own realtime threads. The frame callback does only two cheap things: report a level, and hand the frame to a bounded channel. Everything expensive lives on a dedicated notetaker-transcribe thread — mixing, resampling, archiving to WAV, and Whisper inference. Inference is therefore not on the UI thread, and not on the audio thread either, so a slow decode can never stall the meters or drop audio.

Meters are driven straight from the frame callback for that reason: they stay smooth while a decode is in flight. The recording timer is wall-clock time, not audio time, so it cannot drift when the decoder falls behind.

Where "me" and "them" come from

The microphone and the system output are captured as separate streams. That gives a cheap speaker attribution without a diarisation model: a segment is labelled You when the mic was dominant and Them when the system output was. Overlapping speech falls back to Speaker. Full diarisation is out of scope.

How streaming works despite Whisper not being a streaming model

  1. Every frame is buffered. The cheap energy VAD only decides where utterances are cut, never what is kept — gating the buffer on it is what used to drop quiet speech.
  2. While speech continues, the accumulated audio is decoded periodically into a partial line, shown greyed out and then replaced.
  3. When a run of silence follows, or the utterance grows past a ceiling, the buffer is trimmed to the last voiced sample and decoded once more as a final line, which is what gets persisted. Chunks containing no voiced audio are discarded rather than sent to the decoder, since decoding pure silence only invites hallucinations.

The level recorded per frame is accumulated over the utterance, and that is what picks the speaker for the final.

Building

Prerequisites

  • Rust 1.80 or newer.
  • A C/C++ toolchain (gcc/clang), pkg-config, and cmake — required by libsqlite3-sys, whisper.cpp, and TLS crates.
  • Linux: PipeWire development headers. libclang is only needed if you let whisper-rs-sys generate its bindings; see below.

PipeWire version note. libspa-sys generates its bindings from your installed SPA headers at build time, so the crate version must understand those headers. This project uses pipewire 0.10, which handles both the 0.3.x and 1.x layouts. Older revisions (0.8 and earlier) fail against PipeWire 1.x with no field `data` on type `spa_pod_builder` . If you need APIs newer than your default feature set, enable the matching feature, e.g. pipewire = { version = "0.10", features = ["v1_0_0"] }.

Debian/Ubuntu:

sudo apt install build-essential pkg-config cmake libpipewire-0.3-dev libclang-dev

Fedora:

sudo dnf install gcc-c++ make pkgconf-pkg-config cmake pipewire-devel clang-devel

sqlite3 itself is not required: the rusqlite bundled feature compiles SQLite from source.

Building and running

The native backends are behind feature flags, so the crate always builds even on a machine without PipeWire or Whisper toolchains:

# UI only: mock capture and mock transcription (no native deps beyond the GUI).
cargo run

# Real capture + local transcription.
cargo build --release --features pipewire,whisper
cargo run   --release --features pipewire,whisper

Why WHISPER_DONT_GENERATE_BINDINGS=1

whisper-rs-sys runs bindgen over its vendored whisper.h at build time, which requires libclang and is sensitive to the installed toolchain. On some setups bindgen silently emits an opaque whisper_full_params (a lone _address field) instead of the real struct, and the build then fails with messages like:

error[E0609]: no field `n_threads` on type `whisper_full_params`
   = note: available field is: `_address`

The project's .cargo/config.toml sets WHISPER_DONT_GENERATE_BINDINGS=1, which makes the crate skip bindgen and use the bindings its author generated from the same vendored sources. That fixes the build and drops the libclang requirement, and it keeps Whisper on the CPU, which is what we want anyway so it does not contend with the UI's GPU renderer.

Two things to know:

  • Force one rebuild if whisper-rs-sys has already run. The crate does not declare this variable via cargo:rerun-if-env-changed, so cargo will keep reusing a cached bad bindings.rs. Run cargo clean -p whisper-rs-sys once (or cargo clean).
  • Opting out. The crate tests for the variable's presence, so setting it to 0 would not disable it — comment the line out instead. You need to do this if you enable whisper-rs/vulkan, because the pre-generated bindings contain no Vulkan entry points.

Feature matrix

Feature Default Effect when off
ui yes The binary is not built; the library still compiles.
pipewire no Capture falls back to the mock backend; capture_is_available() is false.
whisper no Transcription falls back to the mock engine.

With everything off you get a fully working application that uses synthetic audio and placeholder transcripts. That is intentional: the pipeline, database, and UI stay testable on any machine.

Tests

cargo test --lib                      # core: models, db, audio, chunker, LLM, prompts
cargo test --lib --all-features       # additionally exercises the real backends

Using it

  1. Launch the app and press New meeting.
  2. Press Record and talk. The transcript streams in; type your own notes alongside.
  3. Press Stop. The mixed audio is archived and the transcript is committed.
  4. Press Summarise. The transcript and your notes are sent to the configured model.
  5. Export writes the summary and your notes to Markdown under the data directory.

Summaries and API keys

Open Settings and pick a provider. Keys can be typed in, but the environment is always preferred and keeps secrets out of the config file:

export OPENAI_API_KEY=...
export ANTHROPIC_API_KEY=...
export OPENROUTER_API_KEY=...
Provider Models are named Notes
OpenAI gpt-4o, gpt-4o-mini Sends response_format: json_object.
Anthropic claude-sonnet-4-5, … Uses the Messages API.
OpenRouter openai/gpt-4o, anthropic/claude-…, … One key for many vendors' models. Model ids are namespaced vendor/model.
Mock (offline) — No key, no network.

OpenRouter does not send response_format: json_object, because only some upstream models honour it and others reject the request outright. It relies on the prompt and the tolerant response parser instead, so almost any model listed on OpenRouter will work — including the cheap ones, which is handy for the live pass.

The Mock (offline) provider needs no key and performs real (if simple) extraction: - [ ] checkboxes and TODO:/action: lines become action items, and decided: lines become decisions. This keeps the app useful and testable without a network.

Models

Whisper models are downloaded on demand into the data directory, not bundled. The default is large-v3-turbo (near-large-v3 quality, much lower latency). For a first run, base or small downloads far faster. Choose one in Settings and press Download model. Downloads go to a .part file and are renamed only on success.

Accuracy, for reference:

Model Size Notes
tiny / base 75–142 MB Fine for a smoke test; noticeably lossy on real meetings.
small 466 MB A clear step up, still quick. A good everyday choice.
medium 1.5 GB Better again, slower on CPU.
large-v3-turbo 1.6 GB Default. Large-v3 quality at much lower latency.
large-v3 3.1 GB Best accuracy, heaviest.

Two practical notes:

  • Changing the model requires pressing Save settings, then Download model — the running decoder is built from saved config when you press Record.
  • Accuracy is limited by more than the model: the energy VAD splits utterances at stt.silence_end_ms (default 700 ms), and each partial decode re-reads the growing buffer. The segmenter is tunable in config.json — raise stt.partial_interval_ms (default 1200) if the same text keeps being re-decoded, raise stt.max_chunk_ms (default 8000) so long monologues run longer before a forced final, lower stt.silence_db (default −70) if quiet speech is treated as silence, and adjust stt.min_speech_ms (default 300) to change how short a blip must be to count as noise. These are read when recording starts.

When built without --features whisper, the mock engine is used and no download is needed.

Data locations

Everything stays on the machine, under the platform data directory (~/.local/share/notetaker on Linux):

notetaker.db        meetings, transcripts, notes, summaries
audio/              one WAV per meeting
models/             downloaded Whisper models
exports/            Markdown exports

Config lives at ~/.config/notetaker/config.json.

GPU contention

Iced renders the UI with wgpu, and Whisper can also use a GPU. The per-decoder Use GPU setting is off by default so Whisper stays on the CPU and the two do not compete.

There is no runtime renderer switch: in Iced 0.13 the renderer is selected at compile time by feature flags, and the default build uses wgpu with an automatic software fallback. To force software rendering and leave the GPU entirely free, build with wgpu disabled:

# Cargo.toml
iced = { version = "0.13", default-features = false, features = ["tiny-skia", "tokio", "image"] }

Troubleshooting a recording that produces nothing

Run with logging on, then press Record and speak for a few seconds:

RUST_LOG=notetaker=debug cargo run --release --features pipewire,whisper

The messages appear in order, and whichever one is missing tells you where it stops:

Expected log line Meaning if absent
starting capture via PipeWire … Recording never started — check for an error banner in the UI.
transcription worker ready (…) The worker thread failed to start (the message says which engine).
system audio linked: … Hz, … channel(s), format … The stream never linked to the sink's monitor.
microphone linked: … The stream never linked to the default source.
… first frame received (N samples, …) Linked, but no audio ever arrived.
transcription is falling behind; N audio frames dropped Inference is slower than real time; use a smaller model or raise partial_interval_ms.
audio level X dB (silence threshold Y dB), speech=… Emitted about once a second. If X never rises above Y, the VAD never opens and nothing is transcribed — lower stt.silence_db in the config fil 5B9C e.

Two useful cross-checks while it runs:

  • Do the level meters move? They are driven directly by captured audio, so live meters with no transcript point at transcription, while dead meters point at capture.
  • Is the archive growing? ls -l ~/.local/share/notetaker/audio/ shows the WAV being written; its size should increase by roughly 32 KB per second while recording.

If the log shows a negotiated format other than F32LE, capture will refuse to interpret it and say so — that means the graph ignored our format request and the sample reader needs extending.

Limits and known gaps

  • Speaker attribution is two-way, not true diarisation. Overlapping speech is labelled Speaker.
  • VAD is energy-based. whisper.cpp's Silero VAD is the natural upgrade and is what the ChunkerConfig is shaped around.
  • Live summaries are not wired to the UI yet. LlmConfig::live_summary and the two-tier model fields exist, and Summarizer is cheap enough to call on a timer, but only the end-of-meeting pass is currently triggered.
  • Capture is Linux/PipeWire only. macOS (ScreenCaptureKit) and Windows (WASAPI loopback) backends belong behind the same CaptureBackend trait and are not written.
  • Packaging is unhandled. There is no installer, and as a Flatpak the app would need explicit PipeWire socket permissions to capture.

Verifying a build

cargo fmt --check
cargo clippy --all-targets --features pipewire,whisper -- -D warnings
cargo test --lib --all-features

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages

0