MiniLake is a local emulator of the Databricks REST API, for developing and testing
databricks-sdk, CLI and Terraform code without a workspace.
SQL statements run on DuckDB. EXTERNAL Delta tables are Parquet files with a _delta_log/,
and writes to them go through a generated Spark job. Jobs run spark-submit in a sibling
Docker container. Everything else in the API is either stored and read back, or answers
501 NOT_IMPLEMENTED. It is meant for one developer on localhost, not as a shared server.
Website: https://dmux.github.io/minilake/
pip install minilake
minilake --port 8000
# or
docker run -p 8000:8000 ghcr.io/dmux/minilake:latest
curl http://localhost:8000/_minilake/healthThe web SQL editor is at http://localhost:8000/ui/. Any token is accepted. The image downloads nothing at runtime, so it also works offline (details).
from databricks.sdk import WorkspaceClient
w = WorkspaceClient(host="http://localhost:8000", token="dev")
w.catalogs.create(name="sales")
w.schemas.create(name="store", catalog_name="sales")
wh = w.warehouses.create(name="dev", cluster_size="Small")
def sql(statement):
return w.statement_execution.execute_statement(warehouse_id=wh.id, statement=statement)
sql("CREATE TABLE sales.store.orders (id BIGINT, customer VARCHAR)")
sql("INSERT INTO sales.store.orders VALUES (1, 'Ana')")
print(sql("SELECT * FROM sales.store.orders").result.data_array) # [[1, 'Ana']]- Getting started
- Configuration: environment variables, persistence, TLS
- Databricks SDK
- Terraform and Asset Bundles
- Spark and Delta Lake
- Web UI
- MCP server: optional,
MINILAKE_MCP=1, for LLM agents - Testing and development
- FEATURES.md: status of every service, with design notes
- API coverage: every SDK endpoint probed through the
databricksCLI - Emulation roadmap
These are deliberate. MiniLake targets one developer running it locally.
- No authentication. Any token is accepted. SCIM users and personal access tokens are records, not credentials.
- No access control. The Permissions and Grants APIs store and return what you give them, and nothing is enforced. A test that passes here says nothing about grants in a real workspace.
- The SQL dialect is DuckDB's, not Spark SQL.
- Clusters are a state machine. There is no Spark on them; Spark runs through Jobs.
- MANAGED tables are DuckDB tables with no files, so Spark cannot see them. Use EXTERNAL Delta tables when Spark has to read the data.
- Single process. DuckDB is single-writer, so concurrent load serializes on a lock.
- Not implemented, returns 501: Repos, DLT, Model Registry, Vector Search, Lakeview and Genie, Delta Sharing, and non-Python notebook languages.
- Uneven test coverage.
jobs.py,sql_statements.pyandunity_catalog.pyare tested mostly on the happy path.
MiniLake does not reproduce every quirk of the real service. Tests here run against the
real databricks-sdk, and the Terraform provider and databricks bundle are exercised
end to end, but a green run locally is not a substitute for a run against Databricks.
See CONTRIBUTING.md.
MIT. See LICENSE.