Free, open-source local Databricks emulator for offline development and testing.
Real SQL & Spark execution · Unity Catalog hierarchy · Databricks SDK compatible · Terraform compatible · MIT licensed
GitHub · Container Image (GHCR)
MiniLake is a free, local Databricks API emulator — a single-developer tool for testing databricks-sdk/Terraform code against real SQL, real Delta Lake, and real Job execution, without paying for cloud compute.
| Service | Status | Notes |
|---|---|---|
| Unity Catalog (catalogs, schemas, tables, volumes) | ✅ Real | Each catalog = its own DuckDB database (ATTACH), native catalog.schema.table addressing |
| EXTERNAL Delta Tables | ✅ Real | Real Delta files; INSERT/UPDATE/DELETE via a generated Spark job, reads via delta_scan() |
| SQL Statement Execution | ✅ Real | Real DuckDB; JSON_ARRAY/ARROW_STREAM/CSV, INLINE/EXTERNAL_LINKS |
| SQL Warehouses | ✅ Real | Full CRUD + lifecycle |
| Jobs | ✅ Real | Sibling Docker container execution (Spark) or subprocess fallback; real DAG scheduling (depends_on/run_if); sql_task.file |
| Workspace | ✅ Real | File-backed notebook/script storage (SOURCE/PYTHON only) |
| DBFS & Files API | ✅ Real | File-backed storage, chunked upload |
| Secrets | ✅ Real | Real CRUD; values only resolvable inside job env vars, never via direct API (matches real Databricks) |
| Clusters | ✅ Real state machine | CRUD + timed lifecycle transitions; no real Spark compute (by design) |
| Permissions | ✅ Real CRUD | Single-user "allow-all" default (by design — see Gaps) |
| Identity (SCIM) | ✅ Static | Fake current-user endpoint |
Persistence (MINILAKE_PERSIST=1) |
✅ Real | JSON snapshot on shutdown, restored on startup |
| JupyterLab + PySpark + Delta (optional) | ✅ Real | docker compose --profile notebook up |
| Secrets ACLs, Repos/Git, multi-language notebooks, DBT/pipeline tasks, Model Registry, Vector Search, Dashboards | 🚫 Not implemented | Returns 501 NOT_IMPLEMENTED |
These are deliberate, not oversights — minilake targets one developer running it locally, not a shared/multi-tenant server:
- No real authentication — any token is accepted; there's only ever one real user.
- No access-control enforcement — Permissions API is real CRUD but always allow-all.
- No real Spark compute for Clusters — state machine only; real compute happens via Jobs' sibling containers instead.
- Single process, no HA — and DuckDB's single-writer model means concurrent load from multiple simulated "users" will contend on locks.
- Uneven test coverage — ~45% overall;
jobs.py,sql_statements.py,unity_catalog.pyare covered mostly on happy paths, not edge cases. - Secrets ACLs not implemented — scope/secret CRUD is real, ACL endpoints aren't.
# Option 1: PyPI (simplest)
pip install minilake
minilake --port 8000
# Runs on http://localhost:8000
# Option 2: GitHub Container Registry
docker run -p 8000:8000 ghcr.io/dmux/minilake:latest
# Option 3: Clone and build
git clone https://github.com/dmux/minilake
cd minilake
docker compose up -d
# Verify (any option)
curl http://localhost:8000/_minilake/healthThat's it. No account, no API key, no sign-up.
Minilake exposes internal endpoints for test automation:
# Health check — returns service status
curl http://localhost:8000/_minilake/health
# Readiness check — ensures DuckDB pool and data dir are ready
curl http://localhost:8000/_minilake/ready
# Reset all state — wipe every service back to empty (useful between test runs)
curl -X POST http://localhost:8000/_minilake/reset
# List enabled services
curl http://localhost:8000/_minilake/servicesThe reset endpoint is especially useful in CI pipelines and test suites — call it in setUp/beforeEach to get a clean environment for every test without restarting the container. Note that physical tables and volumes are preserved.
To set up configuration, use environment variables at startup:
docker run -p 8000:8000 \
-e MINILAKE_SERVICES=unity_catalog,sql_statements,sql_warehouses \
ghcr.io/dmux/minilake:latestBy default, minilake only prints its startup banner (ASCII logo + port + enabled services) and stays quiet otherwise — no per-service load logs, no per-request access logs, no SQL/job execution traces. Set MINILAKE_VERBOSE=1 to get full INFO-level logs instead:
docker run -p 8000:8000 \
-e MINILAKE_VERBOSE=1 \
ghcr.io/dmux/minilake:latestBy default, all state (catalogs, jobs, secrets, clusters, ...) lives in memory and is lost when the container stops — real files (Workspace, DBFS, EXTERNAL Delta data) always survive, but metadata doesn't. Set MINILAKE_PERSIST=1 to save a JSON snapshot on shutdown and restore it on the next startup, as long as MINILAKE_DATA_DIR points at a volume that survives the restart too:
docker run -p 8000:8000 \
-e MINILAKE_PERSIST=1 \
-v minilake_data:/data \
ghcr.io/dmux/minilake:latestBy default, notebook_task/spark_python_task Jobs run for real in a sibling Docker container (Apache Spark image), the same pattern LocalStack uses for Lambda. This needs the host's Docker socket mounted:
docker run -p 8000:8000 \
-v /var/run/docker.sock:/var/run/docker.sock \
ghcr.io/dmux/minilake:latestIf a Docker socket isn't available (e.g. restricted CI runners), set MINILAKE_JOB_EXECUTOR=subprocess to fall back to running task scripts as a plain local subprocess instead — no Docker required, at the cost of not matching the real Spark runtime exactly.
from databricks.sdk import WorkspaceClient
# All clients use the same endpoint and any dummy token
w = WorkspaceClient(
host="http://localhost:8000",
token="dev"
)
# Unity Catalog — Catalogs and Schemas
w.catalogs.create(name="my_catalog")
w.schemas.create(name="my_schema", catalog_name="my_catalog")
# Unity Catalog — Native Table Creation
# Create tables that are instantly queryable
w.tables.create(
full_name="my_catalog.my_schema.my_table",
columns=[
{"name": "id", "type_text": "INTEGER"},
{"name": "name", "type_text": "VARCHAR"}
],
table_type="MANAGED"
)
# SQL Warehouses
warehouse = w.warehouses.create(
name="test_warehouse",
cluster_size="Small"
)
w.warehouses.start(warehouse.id)Terraform — point your provider block to the local instance:
provider "databricks" {
host = "http://localhost:8000"
token = "dev"
}
resource "databricks_catalog" "sandbox" {
name = "sandbox"
comment = "Local sandbox catalog"
}
resource "databricks_schema" "things" {
catalog_name = databricks_catalog.sandbox.name
name = "things"
}
resource "databricks_sql_endpoint" "compute" {
name = "local-compute"
cluster_size = "Small"
}For interactive exploration, an optional JupyterLab profile shares minilake's data volume and comes preconfigured with Delta Lake:
docker compose --profile notebook up -d
# open http://localhost:8888 — a quickstart notebook is pre-loadedIt demonstrates registering an EXTERNAL Delta table in minilake, writing to it with real PySpark (df.write.format("delta")), and reading the same data back through minilake's SQL Statement Execution API — proving DataFrame writes and SQL reads see the same real data.
# Run all tests locally
pytest tests/ -v
# Run tests in Docker
bash scripts/run-tests-docker.sh
# Run tests with coverage
pytest tests/ --cov=minilake --cov-report=html- Every push/PR to
mainrunsci.yml: lint (ruff) + the full test suite in Docker. - Tagging a release builds and publishes automatically — no manual Docker build/push needed:
git tag vX.Y.Z
git push origin vX.Y.ZThis triggers release.yml, which re-runs the test suite as a gate, then builds a multi-arch (amd64/arm64) image and publishes it to GitHub Container Registry as ghcr.io/dmux/minilake:X.Y.Z, ghcr.io/dmux/minilake:X.Y, and ghcr.io/dmux/minilake:latest, and creates a GitHub Release with auto-generated notes. Tags must match vX.Y.Z (semver).
The badges at the top of this README (release version, CI status, release status) are live shields.io badges that query the GitHub API directly — they always reflect the latest tag/workflow run with no manual README edit required after a release.
See CONTRIBUTING.md for the project structure, how to add a new API group, and the PR checklist.