Releases

@dmux

Job tasks can now install Python packages (libraries[].pypi) and every job gets DATABRICKS_HOST / DATABRICKS_TOKEN, so WorkspaceClient() works inside a script with no arguments. See the changelog for details.

pip install minilake==1.9.3
docker pull ghcr.io/dmux/minilake:1.9.3

PyPI: https://pypi.org/project/minilake/1.9.3/
Container image: ghcr.io/dmux/minilake:1.9.3

What's Changed

feat: install PyPI libraries for job tasks and give jobs API access by @dmux in #96
chore(release): bump version to 1.9.3 by @dmux in #97

Full Changelog: v1.9.2...v1.9.3

@avison9

Lets browser apps call minilake directly (configurable CORS origins), serves the Jobs API under /api/2.0 and /api/2.1 for older SDKs and CLIs, and fixes jobs/update dropping omitted settings. See the changelog for details.

Thanks to @avison9 for #69. 🙏

pip install minilake==1.9.2
docker pull ghcr.io/dmux/minilake:1.9.2

PyPI: https://pypi.org/project/minilake/1.9.2/
Container image: ghcr.io/dmux/minilake:1.9.2

What's Changed

docs(site): trim landing page copy, stress open source, show contributors by @dmux in #91
fix(jobs): keep omitted settings unchanged on jobs/update by @avison9 in #69
feat: configurable CORS origins and /api/2.0, /api/2.1 jobs aliases by @dmux in #93
chore(release): bump version to 1.9.2 by @dmux in #94

Full Changelog: v1.9.1...v1.9.2

@eddiethedean

A bug-fix release for SQL, Unity Catalog, DBFS/Files, jobs, clusters, SCIM and MCP. See the changelog for details.

Thanks to @eddiethedean for most of the fixes in this release, and to @avison9 for their first contribution (#68). 🙏

pip install minilake==1.9.1
docker pull ghcr.io/dmux/minilake:1.9.1

PyPI: https://pypi.org/project/minilake/1.9.1/
Container image: ghcr.io/dmux/minilake:1.9.1

What's Changed

docs(site): bump version badge to v1.9.0 by @dmux in #30
Fix external table SQL rewriting by @eddiethedean in #84
Preserve SQL types in Arrow results by @eddiethedean in #83
Fix DBFS and Files with relative data directories by @eddiethedean in #82
Fix issue 44: terminate SQL task runs on file errors by @eddiethedean in #81
fix: apply SCIM patches atomically by @eddiethedean in #80
fix(sql): preserve duplicate Arrow result columns by @eddiethedean in #54
[BUG] Preserve snapshots when DBFS uploads are unfinished by @eddiethedean in #59
[BUG] Resolve Files and DBFS paths to UC volume storage by @eddiethedean in #72
[BUG] Honor overwrite=false for chunked DBFS uploads by @eddiethedean in #66
fix(duckdb_pool): apply MINILAKE_DUCKDB_MEMORY_LIMIT to every DuckDB connection by @avison9 in #68
fix: prevent stale cluster transitions from overriding newer requests by @eddiethedean in #79
[BUG] Restore persisted jobs and runs by numeric ID by @eddiethedean in #73
[BUG] Preserve newest results when the SQL statement cache overflows by @eddiethedean in #67
[BUG] Preserve DML predicates when comments mention DDL clauses by @eddiethedean in #62
[BUG] Preserve existing DBFS move destinations by @eddiethedean in #65
[BUG] Honor idempotency tokens for jobs/run-now by @eddiethedean in #64
[BUG] Reject invalid job dependency graphs by @eddiethedean in #70
[BUG] Reject SQL statements with invalid namespaces by @eddiethedean in #61
[BUG] Preserve existing table when seed replacement fails by @eddiethedean in #63
[BUG] Clear default DuckDB data on full reset by @eddiethedean in #71
[BUG] Honor force when deleting catalogs and schemas by @eddiethedean in #60
fix: clear cached workspace directory metadata by @eddiethedean in #78
fix: cascade cleanup when deleting schemas and catalogs by @eddiethedean in #77
fix: execute saved queries in their namespace by @eddiethedean in #76
fix: keep cluster edits atomic on invalid references by @eddiethedean in #75
fix: preserve names on idempotent updates by @eddiethedean in #74
chore(deps): update Python and UI dependencies within current constraints by @dmux in #86
fix(docker_executor): explain the missing Docker daemon at startup by @dmux in #87
chore(release): bump version to 1.9.1 by @dmux in #88

New Contributors

@avison9 made their first contribution in #68

Full Changelog: v1.9.0...v1.9.1

@eddiethedean

pip install minilake==1.9.0
docker pull ghcr.io/dmux/minilake:1.9.0

PyPI: https://pypi.org/project/minilake/1.9.0/
Container image: ghcr.io/dmux/minilake:1.9.0

What's Changed

fix(sql): preserve literals when rewriting backtick identifiers by @eddiethedean in #25
chore(deps): update dependencies to resolve Dependabot security alerts by @dmux in #27
chore(release): bump version to 1.9.0 by @dmux in #28

New Contributors

@eddiethedean made their first contribution in #25

Full Changelog: v1.8.0...v1.9.0

@dmux

A dependency security release: it clears every open Dependabot alert. There are no functional changes.

pip install minilake==1.8.0
docker pull ghcr.io/dmux/minilake:1.8.0

Security

All of these are transitive dependencies of the web UI, fixed with a lockfile-only refresh of ui/pnpm-lock.yaml. Each patched version already fits the range its parent declares, so ui/package.json is unchanged and no overrides were needed.

Package	Before	After	Advisories
fast-uri	3.1.5	3.1.8	4 high: host confusion and SSRF via URI normalization
js-yaml	4.3.1	4.3.2	1 high: merge-key CPU exhaustion
hono	4.13.3	4.13.11	3 medium: `toSSG()` path escape, `parseBody()` memory exhaustion, fragment query parsing
qs	6.15.3	6.16.0	2 medium: DoS via `isBuffer`, array-limit bypass
undici	7.29.0	7.30.0	1 medium: WebSocket permessage-deflate DoS
ip-address	10.5.0	10.7.2	2 medium: link-local / NAT64 classification bypass

Exposure was low. These packages come in through shadcn (its MCP SDK, express and ajv) and eslint, not through the exported UI bundle that ships in the wheel. pnpm audit now reports no known vulnerabilities.

Per-release detail lives in CHANGELOG.md.

What's Changed

chore(release): 1.8.0 — fix open Dependabot alerts by @dmux in #21

Full Changelog: v1.7.8...v1.8.0

PyPI: https://pypi.org/project/minilake/1.8.0/
Container image: ghcr.io/dmux/minilake:1.8.0

@dmux

Makes the Terraform claim testable, then fixes what the test found.

pip install minilake==1.7.8
docker pull ghcr.io/dmux/minilake:1.7.8

Coverage: 208 → 267 of the 1168 endpoints the databricks-sdk calls. CLI command
groups working end to end: 21 → 30.

The measure changed, and it matters

That raw figure is 22.9%, and the roadmap used to quote it without qualification. It is
the wrong number. 604 of the missing endpoints are in blocks this project has
deliberately decided not to emulate — MLflow, the Account API, identity v2,
marketplace, Genie, Lakebase, vector search, serving, clean rooms, apps.

Against the reachable surface, coverage is 267 of 564 — about 47%. That is what
docs/EMULATION_ROADMAP.md
now leads with. Counting deliberate scope cuts as gaps makes progress look permanently
hopeless and hides which gaps are real.

Terraform is now tested, not asserted

docs/terraform.md promised the Databricks provider worked against minilake. Nothing
verified it — there was no Terraform test in the repo at all, and the services added in
1.7.7 (SCIM, tokens, cluster policies, instance pools) were specifically
Terraform-facing resources with no way to check them.

tests/terraform/ now drives the real provider through apply → plan → destroy
over a 15-resource config and asserts the second plan is empty. terraform and a pinned
provider mirror are baked into the test image, so the suite still needs no network.

Its first run found three defects that every SDK-driven test in the suite had missed,
because they live in behaviour only the provider exercises:

Defect	Why no SDK test caught it
The provider posts `"entitlements": [{}]` for a group with none; minilake echoed the empty entry back and broke the provider on its own input (`Invalid address to set: []string{""}`)	The SDK never sends `[{}]`. Real Databricks drops valueless entries from multi-valued SCIM attributes; now so does minilake
SQL warehouses never reported `auto_stop_mins`, `enable_photon`, `max_num_clusters` or `odbc_params`, so every `terraform plan` showed a permanent in-place update	The SDK reads back only what it asked about; Terraform diffs the whole object
SCIM ignored the `attributes` projection the provider requests	The SDK does not send it

Permanent drift is worse than a 501: it looks like it works until someone runs plan
twice. The empty-plan assertion is what catches it.

New endpoints

Unity Catalog grants — w.grants.* and Terraform's databricks_grants, with real
inheritance: a grant on a catalog is effective on its schemas and their tables, and
each effective privilege is tagged with the ancestor it came from. Without that,
get_effective would just be get under another name.
The workspace-admin APIs — git credentials, IP access lists, global init scripts,
notification destinations, instance profiles and workspace-conf. Each is trivial
alone; together they are what a realistic terraform apply touches before it reaches
a catalog, where a single 501 fails the whole plan.
Legacy /api/2.0/preview/sql/* — queries, alerts, dashboards, widgets,
visualizations and data sources, as adapters over the modern stores rather than a
second copy. An object created through either surface is visible and editable from the
other.

Three details that are load-bearing

The legacy list endpoints must paginate. The SDK walks them by incrementing page
until a response comes back empty, so a handler that ignores the parameter never
terminates. The symptom is a hung client, not an error.

DELETE trashes and POST .../trash/{id} restores, not the other way round.
Implemented backwards, a client asking to restore would destroy.

securable_type arrives as SecurableType.CATALOG, not CATALOG. The SDK's enum is
not a str subclass, so its f-string interpolation puts the repr on the wire. Both forms
are accepted; without that, grants stored under an unrecognised key and inheritance
silently found nothing.

What this does not give you

Nothing added here is enforced. Grants, secret ACLs, IP access lists and cluster policies
are stored and read back faithfully and checked nowhere; no IP is blocked, no init script
runs, no notification is sent. A terraform plan that applies cleanly against minilake
says the shape of your config is right, not that its access control is. docs/terraform.md
now states this plainly.

Also fixed

scripts/cli_coverage.py was scoring all 8 Files API endpoints as missing — they are
implemented and tested. The SDK writes /api/2.0/fs/files{path} with the parameter
carrying its own leading slash, and the normalizer produced a path matching no route.
Those two paths are the only ones in the SDK shaped that way.

Testing

378 passing, up from 338. Running pytest on a host without terraform installed
skips the conformance tests rather than failing.

Per-release detail lives in CHANGELOG.md.

What's Changed

Terraform conformance test, UC grants, workspace-admin belt, legacy SQL by @dmux in #19
chore(release): 1.7.8 — Terraform conformance, grants, admin belt by @dmux in #20

Full Changelog: v1.7.7...v1.7.8

PyPI: https://pypi.org/project/minilake/1.7.8/
Container image: ghcr.io/dmux/minilake:1.7.8

@dmux

Expands API emulation coverage from 13.3% to 17.8% of the endpoints the
databricks-sdk actually calls (155 → 208), and adds the tooling that measures it.
Command groups of the databricks CLI that now work end to end: 10 → 21.

No behaviour changes for existing SQL, Unity Catalog or Jobs workloads.

pip install minilake==1.7.7
docker pull ghcr.io/dmux/minilake:1.7.7

Coverage is now measured, not asserted

scripts/cli_coverage.py points the real databricks CLI at a disposable minilake and
probes every (method, path) pair the installed SDK calls, recording which fall through
to the catchall 501. The unit is the method+path pair rather than the path: probing
everything with GET scores POST-only endpoints as missing, because FastAPI falls through
when no method matches — that distinction alone moved the measured baseline from 10.0% to
13.3%.

It produces docs/CLI_COVERAGE.md
(generated) and docs/EMULATION_ROADMAP.md
(curated — what is worth building next, with measured gap counts). The CLI is pinned into
Dockerfile.test so numbers stay comparable between runs.

Things that already had an engine and only lacked an endpoint

sql_task.query now executes for real. It was SKIPPED. Zero new endpoints were
needed — the Queries API had existed all along, and the "no Queries API" note it was
skipped for was simply stale.
POST /api/2.2/jobs/runs/submit — one-shot runs that carry their tasks inline and
belong to no job, the path databricks bundle run and most CI code take. Reuses the
existing DAG scheduler, and honours idempotency_token so a repeated submit returns
the original run rather than starting a second one.
Alerts (/api/2.0/sql/alerts) and sql_task.alert, which evaluates the
condition against real query results rather than reporting a stored state.

Compatibility walls removed

Unity Catalog metastore reads — current-metastore-assignment, metastore_summary,
and metastores list/get. Small, but load-bearing: several clients call the assignment
during setup and previously gave up at the 501 before ever reaching a catalog that
worked perfectly well.
SCIM Users, Groups and Service Principals (18 endpoints) — CRUD, the PATCH forms
Terraform sends for group membership (including members[value eq "id"]), and
attribute eq "value" filtering. The current user is seeded so Me and Users never
disagree about who is signed in.
Personal access tokens, secret scope ACLs, and clusters update/pin/unpin.
update is a partial edit honouring update_mask, so changing one setting no longer
blanks the rest of the spec the way a full edit would.

New functionality

Unity Catalog functions. A SQL function is created as a real DuckDB MACRO, so a
function registered through the UC API is then callable from the Statement Execution
API by its three-part name — SELECT cat.sch.add_tax(100) actually computes. Not
metadata-only.
Cluster policies and instance pools. Neither is enforced — there is no compute here
for a policy to constrain, and pool stats are permanently zero — but both resolve, so a
bundle or Terraform config naming one deploys instead of hitting a 501.

Behaviour change worth knowing

Clusters now retain and validate policy_id and instance_pool_id.
CreateClusterRequest previously dropped both via extra="ignore". Now that policies and
pools are real resources, a cluster that forgot the policy it was created with would read
as permanent drift in Terraform. A cluster naming a policy or pool that does not exist is
rejected, and one in use cannot be deleted.

What this does not give you

Identities, tokens and ACLs are records, not credentials. minilake accepts any bearer
token by design, so creating a user makes no way to sign in, and revoking a token locks
nobody out. They exist because tooling creates them as a setup step and stops when that
step fails — not because the values mean anything.

Likewise, secret ACLs and cluster policies are stored and read back faithfully but never
enforced. A test that passes here still says nothing about grants in a real workspace.

Also in this release

Docs corrected: README.md and the published site had both described Identity as a
static current-user endpoint and listed secret ACLs as unimplemented. Neither was true
any more.
next bumped to 16.3.3 in the web UI (#16).

Testing

338 tests passing, up from 267 — 71 new, all driven through the real databricks-sdk
against the Dockerized stack, per the project's rule that the official client is the
source of truth.

Per-release detail lives in CHANGELOG.md.

What's Changed

Expand Databricks API emulation: 13.3% -> 17.8% coverage by @dmux in #17
build(deps): bump next from 16.3.2 to 16.3.3 in /ui by @dependabot[bot] in #16
chore(release): 1.7.7 — docs sync and version bump by @dmux in #18

New Contributors

@dependabot[bot] made their first contribution in #16

Full Changelog: v1.7.6...v1.7.7

PyPI: https://pypi.org/project/minilake/1.7.7/
Container image: ghcr.io/dmux/minilake:1.7.7

@dmux

pip install minilake==1.7.6
docker pull ghcr.io/dmux/minilake:1.7.6

PyPI: https://pypi.org/project/minilake/1.7.6/
Container image: ghcr.io/dmux/minilake:1.7.6

What's Changed

fix: VS Code extension login, run status, and workspace file system against minilake by @dmux in #14
chore(release): 1.7.6 by @dmux in #15

Full Changelog: v1.7.4...v1.7.6

Dependency fix and maintenance release. No API changes.

Fixed

pip install minilake / uv add minilake crashed on startup with ModuleNotFoundError: No module named 'httpx'.
httpx is imported unconditionally at runtime — by the JupyterLab reverse proxy (notebook.py) and the MCP ASGI client
(mcp/client.py), and app.py imports notebook.py at module top level — but it was only declared in the dev
dependency group, which pip and uv add never install. This regressed in 1.6.1 when the notebook proxy landed;
before that httpx was test-only. The Docker image and minilake[mcp] were unaffected because the mcp package pulls
httpx transitively. httpx is now a first-class entry in [project.dependencies].

Changed

Python dependencies updated — 18 patch/minor moves via uv lock --upgrade, notably cryptography 50.0.1,
pydantic 2.13.5, mcp 1.29.1 (still pinned <2), websockets 17.1, and the Jupyter stack (jupyter-server 2.21.0,
jupyter-client 8.10.0, ipython 9.17.0).

Full Docker test suite green (266 passed).

pip install minilake==1.7.4
docker pull ghcr.io/dmux/minilake:1.7.4

PyPI: https://pypi.org/project/minilake/1.7.4/
Container image: ghcr.io/dmux/minilake:1.7.4
Full diff: v1.7.3...v1.7.4

@dmux

pip install minilake==1.7.3
docker pull ghcr.io/dmux/minilake:1.7.3

PyPI: https://pypi.org/project/minilake/1.7.3/
Container image: ghcr.io/dmux/minilake:1.7.3

What's Changed

fix: query editor hang, MCP startup warning, and 1.7.3 maintenance by @dmux in #12

Full Changelog: v1.7.2...v1.7.3

v1.9.3

What's Changed

Contributors

Uh oh!

v1.9.2

What's Changed

Contributors

Uh oh!

v1.9.1

What's Changed

New Contributors

Contributors

Uh oh!

v1.9.0

What's Changed

New Contributors

Contributors

Uh oh!

v1.8.0

Security

What's Changed

Contributors

Uh oh!

v1.7.8

The measure changed, and it matters

Terraform is now tested, not asserted

New endpoints

Three details that are load-bearing

What this does not give you

Also fixed

Testing

What's Changed

Contributors

Uh oh!

v1.7.7

Coverage is now measured, not asserted

Things that already had an engine and only lacked an endpoint

Compatibility walls removed

New functionality

Behaviour change worth knowing

What this does not give you

Also in this release

Testing

What's Changed

New Contributors

Contributors

Uh oh!

v1.7.6

What's Changed

Contributors

Uh oh!

v1.7.4

Fixed

Changed

Uh oh!

v1.7.3

What's Changed

Contributors

Uh oh!