8000 minilake/CHANGELOG.md at v1.9.2 · dmux/minilake · GitHub
[go: up one dir, main page]
Skip to content

CHANGELOG.md

Latest commit

 

History

History
558 lines (448 loc) · 28.3 KB

CHANGELOG.md

File metadata and controls

558 lines (448 loc) · 28.3 KB
Raw

Changelog

All notable changes to minilake are recorded here.

The format follows Keep a Changelog, and the project adheres to Semantic Versioning.

Per-feature design rationale and known limitations live in FEATURES.md; this file records what changed between releases.

[1.9.2] — 2026-11-05

Lets browser apps talk to minilake directly, keeps older Databricks SDKs and CLIs working against the Jobs API, and fixes jobs/update dropping settings that were left out of the request.

Thanks to @avison9 for #69.

Added

  • MINILAKE_CORS_ORIGINS — a comma-separated allowlist of browser origins that may call the API cross-origin (* allows any). Unset keeps the same-origin default, and MINILAKE_DEV_CORS still works. Preflights also answer Access-Control-Allow-Private-Network, which Chrome requires when a page calls localhost. The Docker Compose file defaults it to http://localhost:3000 (#92, #93).
  • Jobs API version aliases — /api/2.0/jobs and /api/2.1/jobs are served as aliases of /api/2.2/jobs, so older SDKs and CLIs no longer hit the 501 catch-all (#93).

Fixed

  • jobs/update keeps settings that the request omits instead of resetting them, such as the job name (#69).

[1.9.1] — 2026-10-29

A bug-fix release. It makes SQL, Unity Catalog, DBFS/Files, jobs, clusters and SCIM safer on edge cases, and fixes data-loss and state bugs found by users.

Thanks to @eddiethedean (Odos Matthews) for most of the fixes in this release, and to @avison9 for #68.

Fixed

SQL

  • Invalid namespaces are rejected instead of silently resolving to the wrong schema (#61).
  • Comments that mention DDL clauses no longer change DML predicates (#62).
  • External Delta table references are rewritten safely, including nested queries and write statements (#84).
  • Arrow results keep duplicate column names and SQL types (#54, #83).
  • The newest result is kept when the statement cache overflows (#67).
  • Saved queries run in their own namespace (#76).
  • MINILAKE_DUCKDB_MEMORY_LIMIT applies to every DuckDB connection (#68).
  • Full reset clears the default DuckDB data (#71).

Unity Catalog

  • force is honored when deleting catalogs and schemas, and deleting them cleans up their children (#60, #77).

DBFS, Files and Workspace

  • Files and DBFS paths resolve to Unity Catalog volume storage, and relative data directories work (#72, #82).
  • Moves no longer overwrite an existing destination, and overwrite=false is honored for chunked uploads (#65, #66).
  • Unfinished uploads no longer break persisted snapshots (#59).
  • Cached workspace directory metadata is cleared on change (#78).

Jobs, clusters and SCIM

  • Invalid job dependency graphs are rejected (#70).
  • jobs/run-now honors idempotency tokens, and persisted jobs and runs are restored by numeric ID (#64, #73).
  • SQL task runs end when the SQL file cannot be read (#81).
  • Cluster edits are atomic, and stale transitions no longer override newer requests (#75, #79).
  • Idempotent updates keep existing names (#74).
  • SCIM patches are applied atomically (#80).

MCP

  • seed_table keeps the existing table if the replacement fails (#63).

Changed

  • Clearer startup message when the Docker daemon is missing (#87).
  • Python and UI dependencies updated within current constraints (#86).

[1.9.0] — 2026-10-24

Hardens the backtick-identifier rewriter that lets Databricks-style `name` quoting run on DuckDB, and clears a fresh round of Dependabot alerts.

Thanks to @eddiethedean (Odos Matthews) for the SQL fixes in this release (#25).

Fixed

  • Backtick rewriting no longer corrupts string literals, comments or dollar-quoted tags. When translating Databricks `col` identifiers to DuckDB's "col", the rewriter now skips over single/double/E-string literals, -- and /* */ comments (including continuations) and $tag$ ... $tag$ blocks, matching DuckDB's own Unicode token boundaries instead of mangling text that merely contained a backtick.

Performance

  • Statements with no backtick are passed through untouched, skipping the rewrite pass entirely.

Security

  • Dependabot alerts cleared via transitive dependency bumps (#27).

1.8.0 — 2026-09-29

Clears every open Dependabot alert. The change is only in ui/pnpm-lock.yaml: each patched version fits the range its parent already declares, so ui/package.json is unchanged and no overrides are needed.

Security

  • Transitive UI dependencies bumped within their existing semver ranges: fast-uri 3.1.5 → 3.1.8 (4 high-severity host-confusion / SSRF advisories), js-yaml 4.3.1 → 4.3.2, hono 4.13.3 → 4.13.11, qs 6.15.3 → 6.16.0, plus undici 7.29.0 → 7.30.0 and ip-address 10.5.0 → 10.7.2, which pnpm audit flagged before Dependabot did. All of these come in through shadcn (its MCP SDK, express and ajv) or eslint, not the exported UI bundle, so actual exposure was low. pnpm audit now reports no known vulnerabilities.

1.7.8 — 2026-09-15

Makes the Terraform claim testable, then acts on what the test found.

Coverage: 208 → 267 of the 1168 endpoints the databricks-sdk calls (17.8% → 22.9%). That raw percentage was always the wrong measure, though — 604 of the missing endpoints are in blocks this project deliberately skips. Against the reachable surface it is 267 of 564, about 47%. CLI command groups working end to end: 21 → 30.

Added

  • A Terraform conformance test (tests/terraform/). docs/terraform.md promised the Databricks provider worked against minilake and nothing verified it. The test drives the real provider through apply → plan → destroy over a 15-resource config and asserts the second plan is empty. terraform and a pinned provider mirror are baked into Dockerfile.test, so the suite still needs no network; running pytest on a host without terraform skips these rather than failing.
  • Unity Catalog grants (/api/2.1/unity-catalog/permissions and effective-permissions) — w.grants.* and Terraform's databricks_grants, with real inheritance: a grant on a catalog is effective on its schemas and their tables, and each effective privilege is tagged with the ancestor it came from.
  • The workspace-admin APIs: git credentials, IP access lists, global init scripts, notification destinations, instance profiles and workspace-conf. Each is trivial alone; together they are what a realistic terraform apply touches before it reaches a catalog, where a single 501 fails the whole plan.
  • Legacy /api/2.0/preview/sql/* — queries, alerts, dashboards, widgets, visualizations and data sources, implemented as adapters over the modern stores rather than a second copy. An object created through either surface is visible and editable from the other.

Fixed

Three defects the conformance test found on its first run, all in behaviour only the Terraform provider exercises, which is why every SDK-driven test in the suite had missed them:

  • SCIM echoed back empty multi-valued entries. The provider posts "entitlements": [{}] for a group with none; minilake stored and returned it, and the provider then failed on its own input with Invalid address to set: []string{""}. Valueless entries are now dropped, as the real API does.
  • SQL warehouses caused permanent Terraform drift. auto_stop_mins, enable_photon, max_num_clusters and odbc_params were never reported, so every terraform plan showed an in-place update. They are now stored and returned with the real API's defaults. None of them changes how a statement executes.
  • SCIM ignored the attributes projection. The provider reads a group with ?attributes=displayName,externalId,entitlements; returning the full object instead broke its field-by-field mapping.

Also fixed in the coverage tooling: scripts/cli_coverage.py scored all 8 Files API endpoints as missing. The SDK writes /api/2.0/fs/files{path} with the parameter carrying its own leading slash, and the normalizer produced a path matching no route. Those two paths are the only ones in the SDK shaped that way.

Changed

  • docs/EMULATION_ROADMAP.md now leads with the reachable-surface number and says why the raw percentage understates progress. It previously repeated the raw figure without qualification.
  • docs/terraform.md lists the resource types the conformance test covers, and states plainly that nothing Terraform creates here is enforced — a plan that applies cleanly says the shape of a config is right, not that its access control is.
  • README.md and the published site list grants, workspace admin and legacy SQL.

Notes

Nothing added in this release is enforced. Grants, secret ACLs, IP access lists and cluster policies are stored and read back faithfully and checked nowhere; no IP is blocked, no init script runs, no notification is sent.

Test suite: 378 passing, up from 338.

1.7.7 — 2026-09-15

Expands API emulation coverage from 13.3% to 17.8% of the endpoints the databricks-sdk calls (155 → 208), and adds the tooling that measures it. CLI command groups working end-to-end: 10 → 21. No behaviour changes for existing SQL/UC/Jobs workloads.

Added

  • Coverage tooling. scripts/cli_coverage.py drives the real databricks CLI against a running minilake and probes every (method, path) the installed SDK calls, recording which fall through to the catchall 501. The unit is the method+path pair rather than the path: probing everything with GET scores POST-only endpoints as missing, because FastAPI falls through when no method matches. Outputs docs/CLI_COVERAGE.md (generated) and docs/EMULATION_ROADMAP.md (curated). The CLI is pinned into Dockerfile.test so numbers stay comparable across runs.
  • sql_task.query and sql_task.alert execute for real. Both were SKIPPED. A query resolves the saved query's text through the Queries API — which had existed all along, making the previous "no Queries API" note stale — and an alert re-runs the watched query and evaluates its condition. Only sql_task.dashboard is still SKIPPED.
  • POST /api/2.2/jobs/runs/submit — one-shot runs that carry their tasks inline and belong to no job, the path databricks bundle run and most CI code take. Reuses the existing DAG scheduler, and honours idempotency_token so a repeated submit returns the original run.
  • Alerts API (/api/2.0/sql/alerts) — full CRUD with update_mask. Conditions are evaluated against real query results rather than a stored state.
  • SCIM Users, Groups and Service Principals (/api/2.0/preview/scim/v2/*, 18 endpoints) — CRUD, the PATCH forms Terraform sends for group membership (including members[value eq "id"]), and attribute eq "value" filtering. The current user is seeded so Me and Users never disagree.
  • Unity Catalog metastore reads — current-metastore-assignment, metastore_summary, and metastores list/get. Small, but several clients call the assignment during setup and previously gave up at the 501 before reaching a catalog that worked.
  • Unity Catalog functions — a SQL function is created as a real DuckDB MACRO, so a function registered through the UC API is then callable from the Statement Execution API by its three-part name. Not metadata-only.
  • Personal access tokens (/api/2.0/token/*) — the value is returned exactly once, as in the real API.
  • Secret scope ACLs (/api/2.0/secrets/acls/*) — stored and read back, never enforced.
  • Cluster policies (/api/2.0/policies/clusters/*) and instance pools (/api/2.0/instance-pools/*) — real CRUD. Neither is enforced: there is no compute here for a policy to constrain, and pool stats are permanently zero. They exist so a bundle or Terraform config naming one resolves.
  • Clusters update, pin and unpin — the last gaps in that group. update is a partial edit honouring update_mask, so changing one setting does not blank the rest of the spec.

Changed

  • Clusters now retain policy_id and instance_pool_id. CreateClusterRequest previously dropped them via extra="ignore". Now that policies and pools are real resources, a cluster that forgot the policy it was created with would read as permanent drift in Terraform, so both fields are kept, reported back, and validated on create — a cluster naming one that does not exist is rejected. A policy or pool in use cannot be deleted.
  • README.md and the published site list the services added above. Both previously described Identity as a static current-user endpoint and listed secret ACLs as unimplemented; neither was true after this release's work.

Notes

Identities, tokens and ACLs are records, not credentials. minilake accepts any bearer token by design, so creating a user makes no way to sign in and revoking a token locks nobody out. These exist because tooling creates them as a setup step and stops when that step fails.

Test suite: 338 passing, up from 267.

1.7.6 — 2026-09-08

Fixes four Workspace/Jobs API gaps that broke the Databricks VS Code extension (login, Bundle Resource Explorer, and the Workspace File System browser) and databricks bundle deploy/run against minilake. No behaviour changes for plain SQL/UC workloads.

Fixed

  • bundle deploy/run intermittently failed with "lineage mismatch in state files", even right after a successful deploy. GET /api/2.0/workspace/export always returned the base64-JSON envelope ({content, file_type}), ignoring the direct_download query param. The Databricks CLI reads bundle state files (state/resources.json) with direct_download=true, expecting raw bytes back; the JSON-wrapped response didn't parse as the expected state file, so the CLI silently fell back to "no remote state" (serial=0, lineage="") on every read after the first. workspace/export now streams raw bytes when direct_download=true.
  • The VS Code extension's Bundle Resource Explorer got stuck on "Timeout while fetching run status" for runs that had already finished successfully. runs/get/runs/list never populated run_page_url, which the Databricks CLI prints as Run URL: %s right after triggering a run. Both bundle run's log tailer and the extension's run-status monitor regex-match a run id out of that printed URL to start polling — with no URL, the id was never found, and the extension's 60s watchdog eventually gave up. run_page_url is now populate 8000 d on both endpoints.
  • The extension's Workspace File System browser failed outright with "Can't fetch details for /Users/<user>". /Users/<user> never existed — minilake only ever created /Workspace/Users/<user>/..., as a side effect of a bundle deploy importing files there — and even once a directory existed, get-status/list returned it without an object_id, which the extension treats as a required field on every workspace object. minilake now creates the user's home directory on startup and assigns directories a (lazily cached) object_id, same as files already had.
  • Creating a new notebook/file from the extension's Workspace File System browser failed with "Import format 'AUTO' is not implemented (only SOURCE is supported)". The extension's "New Notebook"/"New File" actions always send format: "AUTO", never SOURCE. workspace/import now accepts AUTO as an alias for SOURCE — minilake only ever stores plain Python source either way, so the two are handled identically.

1.7.4 — 2026-08-31

Dependency fix and maintenance. No API changes.

Fixed

  • pip install minilake / uv add minilake crashed on startup with ModuleNotFoundError: No module named 'httpx'. httpx is imported unconditionally at runtime — by the JupyterLab reverse proxy (notebook.py) and the MCP ASGI client (mcp/client.py), and app.py imports notebook.py at module top level — but it was only declared in [dependency-groups].dev, which pip and uv add never install. This regressed in 1.6.1, when the notebook proxy landed; before that httpx was test-only. The Docker image and minilake[mcp] were unaffected because the mcp package pulls httpx transitively. httpx is now a first-class entry in [project.dependencies].

Changed

  • Python dependencies updated — 18 patch/minor moves via uv lock --upgrade, notably cryptography 50.0.1, pydantic 2.13.5, mcp 1.29.1 (still pinned <2), websockets 17.1, and the Jupyter stack (jupyter-server 2.21.0, jupyter-client 8.10.0, ipython 9.17.0).

1.7.3 — 2026-08-21

Fixes and maintenance. No API changes.

Fixed

  • The Query editor could hang on "Loading…" forever. The page gated its first tab on a hydrated flag set from zustand persist's onRehydrateStorage, which with synchronous localStorage runs inside create() — while the store binding is still in its temporal dead zone. zustand swallowed the resulting ReferenceError, so the flag never flipped and nothing was logged. Anyone with no persisted tabs — a fresh browser, cleared site data, or having just closed the last tab — got a blank page that a reload could not fix. The signal now comes from React (useIsMounted), which cannot get stuck and also makes the first client render agree with the static export's markup.
  • MINILAKE_MCP=1 printed a pydantic-settings warning on startup. FastMCP's own Settings annotates lifespan with a forward reference to FastMCP, which is defined further down the same module, so pydantic-settings >= 2.15 warned about the unresolved reference on every instantiation. The model is now rebuilt once the SDK module is fully imported.

Changed

  • minilake now runs on Python 3.14 in its images and CI, up from 3.11, which left bugfix support in April 2024. requires-python stays at >=3.11: this changes what minilake runs on, not what it can be installed into.
  • Web UI dependencies updated — Next 16.3.2, recharts 3.10.1, lucide-react 1.33, shadcn 4.19, TypeScript 6 — and the transitive dompurify pin lifted to 3.4.14, clearing four DOMPurify advisories reported against the lockfile. Note monaco-editor vendors its own copy of DOMPurify into the bundle actually served, and never calls the APIs those advisories hinge on.
  • Python dependencies updated — twenty patch/minor moves, notably databricks-sdk 0.133.0, starlette 1.6.0, uvicorn 0.52.4 and deltalake 1.6.3.

1.7.2 — 2026-08-21

Dependency fix. No behaviour changes.

Added

  • pytz is now a declared runtime dependency. minilake's own modules use the standard library (datetime.timezone in tls.py, services/files.py, services/saved_queries.py), but code running against an install — notebooks and scripts handling timezone-aware data — reaches for pytz, which nothing in the dependency tree pulled in, so it simply was not there. pip install minilake now brings it.

    Note this covers the Python environment minilake itself is installed into. Job and notebook containers run from their own images (MINILAKE_SPARK_IMAGE and Dockerfile.notebook) and do not install the wheel, so they are unaffected.

1.7.1 — 2026-08-18

Packaging fix. No runtime changes: the 1.7.0 wheel on PyPI is complete and correct, but its sdist was rejected and never published, so 1.7.0 has no source distribution.

Fixed

  • The sdist carried ui/node_modules. hatchling honours only the root .gitignore, so everything ui/.gitignore covers is invisible to it — and the release workflow runs pnpm install before uv build. The sdist came to 350 MB against PyPI's 100 MB per-file limit, and was rejected after the wheel had already been uploaded. Now 7.0 MB.
  • The sdist dropped the web UI, and with it the wheel. uv build builds the sdist first and then builds the wheel from that sdist, but the artifacts override rescuing the gitignored ui_static export was declared only on the wheel target. Nothing failed when this happened — the wheel simply installed with no /ui. uv build --wheel hid it by building straight from the source tree.

Added

  • Two release-workflow guards, both running before anything is uploaded: one fails the publish if the wheel arrives without the web UI, the other if the sdist exceeds PyPI's size limit. A PyPI upload cannot be replaced, so these have to catch it first.

1.7.0 — 2026-08-18

The release that gives minilake a face: an embedded web workspace, and a real JupyterLab inside the image.

Added

Web UI — an Athena-style SQL workspace at /ui, served by minilake itself as a Next.js static export. Same origin as the API, no second process, no extra port.

  • Query editor with multiple tabs, a Monaco editor whose SQL completion is fed by live Unity Catalog metadata, run/cancel/format/EXPLAIN, and an auto-limit that wraps a bare SELECT while leaving explicit LIMITs and DDL alone.
  • Virtualized result grid with column types, sorting, filtering, CSV/JSON download, a chart tab and query statistics.
  • Data catalog tree down to columns, with table details, Preview table and Generate table DDL — plus create and drop for catalogs, schemas, MANAGED tables and volumes. Drops cascade on the server, and the confirmation says so.
  • Jobs: list and run, and now create, edit and delete, with a per-task run detail view alongside the run logs. The editor offers only the task types minilake actually executes (notebook_task, spark_python_task, sql_task.file), since the others are reported SKIPPED rather than faked.
  • Workspace browser over /Workspace — the tree databricks bundle deploy and the Jobs API write into, with a read-only content viewer.
  • Clusters driving the real state machine, and secret scopes and keys. Secret values are never displayed: the API has no read path for them, by design.
  • Warehouses, files, saved queries, recent queries, and a settings page with server health, the signed-in identity and state reset.
  • Light, dark and system themes. Monaco is bundled rather than fetched from a CDN, so the image stays offline-capable.

Embedded JupyterLab — a real notebook server running inside minilake, reverse-proxied at /jupyter and framed by the UI at /ui/notebooks. On by default (MINILAKE_NOTEBOOK=0 disables it).

  • Serving it same-origin is what makes the embedded page possible at all: jupyter-server sends Content-Security-Policy: frame-ancestors 'self', so a notebook server on its own port cannot be framed by the UI. Proxying it satisfies 'self' without weakening the policy.
  • The proxy relays WebSockets, so kernels connect, execute and stream output normally; HTTP responses stream rather than buffer, for JupyterLab's multi-megabyte bundles.
  • The Jupyter process binds 127.0.0.1 and is reachable only through minilake, so it is exposed exactly as far as minilake is and no further.
  • No second Spark in the image. Notebooks reach real Spark through the Jobs API, in the same sibling containers a job uses. The bundled quickstart carries the helper that stages a script, runs it and returns its output.
  • A quickstart notebook is seeded into the notebook directory on first start and never overwritten afterwards, so edits survive an upgrade.
  • New optional extra minilake[notebook], bundled in the Docker image.
  • New settings: MINILAKE_NOTEBOOK, MINILAKE_NOTEBOOK_PATH, MINILAKE_NOTEBOOK_PORT, MINILAKE_NOTEBOOK_DIR.

Query History API — GET /api/2.0/sql/history/queries, backed by the statement cache rather than a second store, so the two cannot drift. Records failed statements, which is usually why the panel gets opened. Supports filter_by.statuses / .warehouse_ids / .statement_ids, max_results with an offset page_token, and include_metrics for the metrics minilake can actually measure.

Saved Queries API — full CRUD at /api/2.0/sql/queries, honouring update_mask and auto_resolve_display_name, and persisted with MINILAKE_PERSIST=1.

Changed

  • POST /_minilake/reset?full=true now honours full. It previously accepted the parameter and ignored it, so the UI's "Full reset" and "Reset state" buttons did the same thing. A full reset deletes the on-disk data directories — the per-catalog DuckDB databases, volume directories and warehouse files that no service reset() touches — while a plain reset still only clears the in-memory registries.
  • Deleting a SQL warehouse from the UI now asks for confirmation, matching every other destructive action.

Fixed

  • Font MIME types and CSS variable loading for the embedded UI: on a bare python:slim image there is no /etc/mime.types, so Python guessed application/octet-stream for hashed .woff2 assets and the browser refused to apply the font.
  • React hydration error #418, and an invalid CSS import.

Removed

  • ConfigItem and ConfigUpdateRequest from admin.py. They modelled a /_minilake/config endpoint that does not exist.

Documentation

  • New docs/ui.md covering the web UI, how it is built and how it is served.
  • FEATURES.md no longer lists query_history and queries among the unimplemented 501 modules — both are implemented and in use by the UI.
  • permissions and dbfs are now recorded as deliberate UI omissions: permissions are stored but never enforced, and DBFS shares its on-disk root with the Files API, which the Files page already browses.

1.6.0 — 2026-08-08

Baseline for this changelog. Core SQL and Unity Catalog with per-catalog isolation, Jobs with real DAG scheduling and Spark execution in sibling containers, Workspace, DBFS, Files, Secrets, Clusters, Permissions, real Spark/Delta round-trips, and the optional MCP server — all working and tested, with MINILAKE_PERSIST wired in.

See FEATURES.md for the full per-feature status of this release.
0