Quint turns speech into clean, semantically-grouped paragraphs — in real time, on a CPU, no GPU. Talk into a microphone (or feed it a live transcript) and Quint transcribes it, restores punctuation, and groups the sentences into coherent paragraphs the instant each one closes — driven by a single-pass streaming chunker built on fast static embeddings. It also ships a batch API for turning podcasts, lectures, and YouTube videos into transcribed, chunked, and summarized text. Open-source and self-hostable.
- Live demo
- Real-time semantic chunking
- How the live demo works
- Batch API
- Quickstart
- License
- Deploy on a GPU cloud
The demo's source and auto-deploy setup live in huggingface-space/.
The heart of Quint is a single-pass streaming chunker (quint/chunking/streaming.py): it reads sentences one at a time, compares each to a running topic centroid, and closes a paragraph the moment the topic drifts. No O(n²) similarity matrix and no lookahead — so it runs on a live stream with only bounded memory.
The split rule is self-calibrating: it opens a new paragraph when a sentence is anomalously dissimilar relative to the running mean/spread seen so far (sim < mean − z·std). The knob z is scale-free (standard deviations), needs no per-corpus tuning, and adapts to each document — the right behaviour for true streaming.
Pair it with Model2Vec static embeddings (~30 MB, no torch, ~80 µs/sentence) for real-time chunking on a CPU:
from model2vec import StaticModel
from quint.chunking.streaming import stream_chunks_live, stream_chunks_adaptive
model = StaticModel.from_pretrained("minishlab/potion-base-8M") # fast static embeddings
def embed(sentence):
return model.encode(sentence)
# Streaming: yields each paragraph the instant it closes (e.g. words from speech-to-text)
for paragraph in stream_chunks_live(sentence_stream, embed):
print(paragraph)
# Batch list-in / list-out, same self-calibrating rule:
paragraphs = stream_chunks_adaptive(my_sentences, embed)stream_chunks(sentences, embed, threshold=...) is also available if you prefer a fixed threshold. The chunker is embedding-agnostic — pass any callable mapping a sentence to a vector (it's L2-normalized internally), so sentence-transformers, OpenAI embeddings, etc. all work.
The mic pipeline in huggingface-space/app.py is a small, fully-local CPU stack — no GPU, no torch:
mic audio → buffer until a pause → Moonshine STT → restore punctuation → semantic chunker → paragraphs
- Speech-to-text: Moonshine (base.en, via sherpa-onnx). Its encoder scales compute with the actual audio length instead of Whisper's fixed 30 s window, so short pause-delimited utterances transcribe in tens of milliseconds.
- Punctuation: a CT-transformer restores sentence punctuation on Moonshine's run-ons (selectively — well-formed sentences pass through untouched), so paragraphs break at real sentence ends.
- Chunking: Model2Vec embeddings + the streaming chunker above.
Quint also runs as a FastAPI service for long-form audio. Once it's running (see Quickstart), interactive docs are at /docs.
| Endpoint | Does |
|---|---|
GET /youtube_transcript?video_id=… |
Fetch a YouTube video's audio and transcribe it |
POST /file_transcript |
Transcribe an uploaded audio file |
POST /chunk { "body": "…" } |
Split long text into semantic paragraphs → { "output": ["…", "…"] } |
POST /best_sentence { "body": "…" } |
Index of the most descriptive sentence |
GET /youtube_summarize?video_id=… |
Chunked per-section summaries of a video |
Install from PyPI (the import package is quint):
pip install quintessentiaFor real-time CPU chunking, also grab the fast static embeddings:
pip install model2vecOr run the batch API locally from source — CPU is fine for chunking and summarization; Whisper transcription is far faster on a GPU (see deploy).
git clone https://github.com/poloniki/quint.git
cd quint
make install # pip install -e .
cp env.sample .env # then set OPENAI_API_KEY
make run_api # serves on http://localhost:8083Then open http://localhost:8083/docs for the interactive API docs.
A small Streamlit frontend lives in frontend/. With the API running:
pip install -r frontend/requirements.txt
streamlit run frontend/app.pySet QUINT_API_URL if the API isn't on http://localhost:8083.
This project is licensed under the MIT License - see the LICENSE file for details.
The real-time chunker and the live demo run anywhere on a CPU. Only the batch Whisper transcription benefits from a GPU. For that, I recommend the JAX backend — it's much faster than the OpenAI-proposed way. See Whisper JAX; one of its tables:
Table 1: Average inference time in seconds for audio files of increasing length. GPU device is a single A100 40GB GPU. TPU device is a single TPU v4-8.
| OpenAI | Transformers | Whisper JAX | Whisper JAX | |
|---|---|---|---|---|
| Framework | PyTorch | PyTorch | JAX | JAX |
| Backend | GPU | GPU | GPU | TPU |
| 1 min | 13.8 | 4.54 | 1.72 | 0.45 |
| 10 min | 108.3 | 20.2 | 9.38 | 2.01 |
| 1 hour | 1001.0 | 126.1 | 75.3 | 13.8 |
Quint runs on any machine with an NVIDIA GPU, so you are free to use whichever cloud provider (AWS, GCP, Azure, Lambda, Paperspace, RunPod, …) or on-prem hardware you prefer. For the best price/performance on transcription, look for an Ada-generation card such as the RTX 6000 Ada or A6000 — these are typically far cheaper than A100-class GPUs while offering comparable CUDA compute capability.
Whatever you pick, you only need an instance that provides:
- An NVIDIA GPU (Ampere/Ada or newer recommended)
- Ubuntu 22.04 (or similar) with CUDA 12 and Docker
- SSH access (root or sudo)
The steps below are provider-neutral: provision the instance however your provider requires, then follow along.
cp env.sample .env # then edit .env
direnv reload # or: source .envSet the following in .env:
| Variable | Used by | Purpose |
|---|---|---|
OPENAI_API_KEY |
API (summarization) | Key for the summarization step |
GPU_TYPE |
API (optional) | Set to A100 to enable bfloat16 on the JAX backend; any other value (or unset) uses float16 |
EMBEDDING_MODEL |
API (optional) | sentence-transformer used for chunking; default all-MiniLM-L6-v2. Use a multilingual model (e.g. paraphrase-multilingual-MiniLM-L12-v2) for non-English text |
SEGMENT_LANGUAGE |
API (optional) | pysbd sentence-split language; default en (e.g. ja for Japanese) |
EMAIL |
deploy helper | Labels / generates your SSH key |
HOST |
deploy helper | Public IP or hostname of your GPU instance |
SSH_USER |
deploy helper | SSH login user for your image (often root, but ubuntu on AWS, your username on GCP, azureuser on Azure) |
Create a GPU instance with your provider using an Ubuntu 22.04 + CUDA 12 + Docker image and your SSH public key. Once it is running, note its public IP (set it as HOST in .env) and connect:
ssh $SSH_USER@$HOST -i ~/.ssh/<your_key>Use the login user your provider specifies for the image.
rootworks on many bare-VM providers, but AWS Ubuntu AMIs useubuntu, GCP uses your username, Azure usesazureuser, etc. Set it asSSH_USERin.env.
The notebook notebooks/Deploy_gpu_instance.ipynb automates the provider-neutral parts: generating an SSH key, copying the code to the host, and building/running the container.
If the instance image already ships with working drivers, skip this. Otherwise run the bundled script on the instance and reboot to load them:
bash scripts/install_nvidia_driver.sh
sudo rebootClone it directly:
git clone https://github.com/poloniki/quint.git
cd quint…or copy your local checkout up with scp (the deploy notebook does this for you).
docker build -t quint --file Dockerfile.jax .
docker run --gpus all -p 80:80 --shm-size=1g --env-file .env quintThe
--env-file .envflag passesOPENAI_API_KEY(and optionalGPU_TYPE) into the container, so make sure.envis present on the instance. Also ensure your provider's firewall / security group allows inbound TCP on port 80 — most clouds only open SSH (port 22) by default.
Your API is now available on the instance's public IP (port 80).