PostgreSQL-orchestrated background threads for Python
pgbg builds on bgt (a great way to reliably run background threads) and adds PostgreSQL integration:
-
A
NOTIFYdispatcher that takes one database connection per process and wakes up an arbitrary number of background threads. -
A table-based leader election with automatic failover. Make sure only one process runs work at a time.
Even with database backing, background tasks are not a job queue like Celery or RQ1. Common use cases include:
- Periodic cleanup duties for expired caches or sessions.
- Maintenance of eventually consistent read models.
- Lightweight background tasks with the transactional outbox pattern.
The core needs and supports only Psycopg 3 for database access.
pgbg comes with optional support for SQLAlchemy in the pgbg.sqlalchemy module that connects everything to a SQLAlchemy Engine.
If you use psycopg-pool, you don't need an adapter at all:
pass your pool's connection method wherever pgbg asks for a connection provider.
Check out our step-by-step tutorial to get an instant feel for the features!
The package is available on PyPI under the pgbg name.
It comes with two optional extras:
sqlalchemy(uv pip install 'pgbg[sqlalchemy]') currently only adds aSQLAlchemy>=2dependency.pool(uv pip install 'pgbg[pool]') installs psycopg-pool.
Full documentation lives at https://pgbg.hynek.me/.
pgbg is written by Hynek Schlawack and distributed under the terms of the MIT license.
The development is kindly supported by my employer Variomedia AG and all my fabulous GitHub Sponsors.
Footnotes
-
But it's useful for implementing worker queues ~ominous foreshadowing~. ↩