Free, open-source local Databricks emulator for offline development and testing.
Real SQL & Spark execution · Unity Catalog hierarchy · Databricks SDK compatible · Terraform compatible · MIT licensed
GitHub · Container Image (GHCR)
MiniLake is a free, local Databricks API emulator — a single-developer tool for testing databricks-sdk/Terraform code against real SQL, real Delta Lake, and real Job execution, without paying for cloud compute.
| Service | Status | Notes |
|---|---|---|
| Unity Catalog (catalogs, schemas, tables, volumes) | ✅ Real | Each catalog = its own DuckDB database (ATTACH), native catalog.schema.table addressing |
| EXTERNAL Delta Tables | ✅ Real | Real Delta files; INSERT/UPDATE/DELETE via a generated Spark job, reads via delta_scan() |
| SQL Statement Execution | ✅ Real | Real DuckDB; JSON_ARRAY/ARROW_STREAM/CSV, INLINE/EXTERNAL_LINKS |
| SQL Warehouses | ✅ Real | Full CRUD + lifecycle |
| Jobs | ✅ Real | Sibling Docker container execution (Spark) or subprocess fallback; real DAG scheduling (depends_on/run_if); sql_task.file |
| Workspace | ✅ Real | File-backed notebook/script storage (SOURCE/PYTHON only) |
| DBFS & Files API | ✅ Real | File-backed storage, chunked upload |
| Secrets | ✅ Real | Real CRUD; values only resolvable inside job env vars, never via direct API (matches real Databricks) |
| Clusters | ✅ Real state machine | CRUD + timed lifecycle transitions; no real Spark compute (by design) |
| Permissions | ✅ Real CRUD | Single-user "allow-all" default (by design — see Gaps) |
| Identity (SCIM) | ✅ Static | Fake current-user endpoint |
Persistence (MINILAKE_PERSIST=1) |
✅ Real | JSON snapshot on shutdown, restored on startup |
| JupyterLab + PySpark + Delta (optional) | ✅ Real | docker compose --profile notebook up |
| Secrets ACLs, Repos/Git, multi-language notebooks, DBT/pipeline tasks, Model Registry, Vector Search, Dashboards | 🚫 Not implemented | Returns 501 NOT_IMPLEMENTED |
These are deliberate, not oversights — minilake targets one developer running it locally, not a shared/multi-tenant server:
- No real authentication — any token is accepted; there's only ever one real user.
- No access-control enforcement — Permissions API is real CRUD but always allow-all.
- No real Spark compute for Clusters — state machine only; real compute happens via Jobs' sibling containers instead.
- Single process, no HA — and DuckDB's single-writer model means concurrent load from multiple simulated "users" will contend on locks.
- Uneven test coverage — ~45% overall;
jobs.py,sql_statements.py,unity_catalog.pyare covered mostly on happy paths, not edge cases. - Secrets ACLs not implemented — scope/secret CRUD is real, ACL endpoints aren't.
# Option 1: PyPI (simplest)
pip install minilake
minilake --port 8000
# Runs on http://localhost:8000
# Option 2: GitHub Container Registry
docker run -p 8000:8000 ghcr.io/dmux/minilake:latest
# Option 3: Clone and build
git clone https://github.com/dmux/minilake
cd minilake
docker compose up -d
# Verify (any option)
curl http://localhost:8000/_minilake/healthThat's it. No account, no API key, no sign-up.
Minilake exposes internal endpoints for test automation:
# Health check — returns service status
curl http://localhost:8000/_minilake/health
# Readiness check — ensures DuckDB pool and data dir are ready
curl http://localhost:8000/_minilake/ready
# Reset all state — wipe every service back to empty (useful between test runs)
curl -X POST http://localhost:8000/_minilake/reset
# List enabled services
curl http://localhost:8000/_minilake/servicesThe reset endpoint is especially useful in CI pipelines and test suites — call it in setUp/beforeEach to get a clean environment for every test without restarting the container. Note that physical tables and volumes are preserved.
To set up configuration, use environment variables at startup:
docker run -p 8000:8000 \
-e MINILAKE_SERVICES=unity_catalog,sql_statements,sql_warehouses \
ghcr.io/dmux/minilake:latestBy default, all state (catalogs, jobs, secrets, cluster
72ED
s, ...) lives in memory and is lost when the container stops — real files (Workspace, DBFS, EXTERNAL Delta data) always survive, but metadata doesn't. Set MINILAKE_PERSIST=1 to save a JSON snapshot on shutdown and restore it on the next startup, as long as MINILAKE_DATA_DIR points at a volume that survives the restart too:
docker run -p 8000:8000 \
-e MINILAKE_PERSIST=1 \
-v minilake_data:/data \
ghcr.io/dmux/minilake:latestBy default, notebook_task/spark_python_task Jobs run for real in a sibling Docker container (Apache Spark image), the same pattern LocalStack uses for Lambda. This needs the host's Docker socket mounted:
docker run -p 8000:8000 \
-v /var/run/docker.sock:/var/run/docker.sock \
ghcr.io/dmux/minilake:latestIf a Docker socket isn't available (e.g. restricted CI runners), set MINILAKE_JOB_EXECUTOR=subprocess to fall back to running task scripts as a plain local subprocess instead — no Docker required, at the cost of not matching the real Spark runtime exactly.
from databricks.sdk import WorkspaceClient
# All clients use the same endpoint and any dummy token
w = WorkspaceClient(
host="http://localhost:8000",
token="dev"
)
# Unity Catalog — Catalogs and Schemas
w.catalogs.create(name="my_catalog")
w.schemas.create(name="my_schema", catalog_name="my_catalog")
# Unity Catalog — Native Table Creation
# Create tables that are instantly queryable
w.tables.create(
full_name="my_catalog.my_schema.my_table",
columns=[
{"name": "id", "type_text": "INTEGER"},
{"name": "name", "type_text": "VARCHAR"}
],
table_type="MANAGED"
)
# SQL Warehouses
warehouse = w.warehouses.create(
name="test_warehouse",
cluster_size="Small"
)
w.warehouses.start(warehouse.id)Terraform — point your provider block to the local instance:
provider "databricks" {
host = "http://localhost:8000"
token = "dev"
}
resource "databricks_catalog" "sandbox" {
name = "sandbox"
comment = "Local sandbox catalog"
}
resource "databricks_schema" "things" {
catalog_name = databricks_catalog.sandbox.name
name = "things"
}
resource "databricks_sql_endpoint" "compute" {
name = "local-compute"
cluster_size = "Small"
}For interactive exploration, an optional JupyterLab profile shares minilake's data volume and comes preconfigured with Delta Lake:
docker compose --profile notebook up -d
# open http://localhost:8888 — a quickstart notebook is pre-loadedIt demonstrates registering an EXTERNAL Delta table in minilake, writing to it with real PySpark (df.write.format("delta")), and reading the same data back through minilake's SQL Statement Execution API — proving DataFrame writes and SQL reads see the same real data.
# Run all tests locally
pytest tests/ -v
# Run tests in Docker
bash scripts/run-tests-docker.sh
# Run tests with coverage
pytest tests/ --cov=minilake --cov-report=html- Every push/PR to
mainrunsci.yml: lint (ruff) + the full test suite in Docker. - Tagging a release builds and publishes automatically — no manual Docker build/push needed:
git tag v1.0.0
git push origin v1.0.0This triggers release.yml, which re-runs the test suite as a gate, then builds a multi-arch (amd64/arm64) image and publishes it to GitHub Container Registry as ghcr.io/dmux/minilake:1.0.0, ghcr.io/dmux/minilake:1.0, and ghcr.io/dmux/minilake:latest, and creates a GitHub Release with auto-generated notes. Tags must match vX.Y.Z (semver).
See CONTRIBUTING.md for the project structure, how to add a new API group, and the PR checklist.