8000 Terraform conformance test, UC grants, workspace-admin belt, legacy SQL by dmux · Pull Request #19 · dmux/minilake · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
10BC8
Diff view
30 changes: 30 additions & 0 deletions Dockerfile.test
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,36 @@ RUN curl -fsSL -o /tmp/databricks.zip \
rm /tmp/databricks.zip && \
databricks --version

# Terraform plus a local mirror of the Databricks provider, for
# tests/terraform/test_terraform_conformance.py. Both are pinned: the conformance test
# is only meaningful if it keeps testing the same provider, and a floating version would
# turn an upstream release into a mysterious CI failure.
#
# `terraform init` normally reaches registry.terraform.io. Warming the plugin cache here
# — where the build already has network — lets the test run `init -plugin-dir` against a
# local mirror instead, so the suite needs no network for this.
ARG TERRAFORM_VERSION=1.16.2
ARG DATABRICKS_PROVIDER_VERSION=1.132.0
# Deliberately NOT TF_PLUGIN_CACHE_DIR at runtime: the test passes this directory to
# `terraform init -plugin-dir`, and if it were also the plugin cache, terraform would try
# to install the provider from the directory into itself and fail. The cache variable is
# set only for the warm-up below, which is what populates the mirror.
ENV MINILAKE_TF_PROVIDER_MIRROR=/opt/tf-providers
RUN curl -fsSL -o /tmp/terraform.zip \
"https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip" && \
python -c "import zipfile; zipfile.ZipFile('/tmp/terraform.zip').extract('terraform', '/usr/local/bin')" && \
chmod 755 /usr/local/bin/terraform && \
rm /tmp/terraform.zip && \
terraform version && \
mkdir -p "$MINILAKE_TF_PROVIDER_MIRROR" /tmp/tf-warmup && \
printf 'terraform {\n required_providers {\n databricks = {\n source = "databricks/databricks"\n version = "%s"\n }\n }\n}\n' \
"${DATABRICKS_PROVIDER_VERSION}" > /tmp/tf-warmup/main.tf && \
cd /tmp/tf-warmup && \
TF_PLUGIN_CACHE_DIR="$MINILAKE_TF_PROVIDER_MIRROR" terraform init -input=false -no-color > /dev/null && \
rm -rf /tmp/tf-warmup && \
find "$MINILAKE_TF_PROVIDER_MIRROR" -name 'terraform-provider-databricks*' | grep -q . && \
chmod -R a+rX "$MINILAKE_TF_PROVIDER_MIRROR"

# Copy project files
COPY pyproject.toml pyproject.toml
COPY src/ src/
Expand Down
78 changes: 77 additions & 1 deletion FEATURES.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@

**minilake** is a local Databricks API emulator backed by DuckDB for real SQL execution. This document details all implemented features, APIs, and their current status.

**Coverage:** 208 of the 1168 endpoints the `databricks-sdk` calls (17.8%), measured — see
**Coverage:** 267 of the 1168 endpoints the `databricks-sdk` calls (22.9%) — or ~47% of
the *reachable* surface, once the blocks this project deliberately skips are excluded. Measured — see
[docs/CLI_COVERAGE.md](docs/CLI_COVERAGE.md) for the generated map and
[docs/EMULATION_ROADMAP.md](docs/EMULATION_ROADMAP.md) for what to build next.

Expand Down Expand Up @@ -877,6 +878,81 @@ plus `GET` / `PUT` / `PATCH` / `DELETE` on `/{id}` for each — 18 routes.

---

### 17d. **Unity Catalog — Grants** ✅

**Module:** `minilake/services/grants.py`
**Endpoints:** `GET`/`PATCH` `/api/2.1/unity-catalog/permissions/{securable_type}/{full_name}`,
`GET /api/2.1/unity-catalog/effective-permissions/{securable_type}/{full_name}`

**Key Features:**

- ✅ Real add/remove deltas; dropping a principal's last privilege drops the principal,
as a real workspace does
- ✅ **Real inheritance.** A grant on a catalog is effective on its schemas and their
tables, and `get_effective` tags each privilege with the ancestor it came from. This
is the whole reason `effective-permissions` is a separate endpoint — without it the
route would look correct while hiding the one behaviour it exists for
- ✅ Rejects a securable whose name does not match its type (a `TABLE` addressed by a
one-part name would sit in the store matching nothing)
- ⚠️ The SDK sends `securable_type` as an `Enum` that is not a `str` subclass, so the
wire carries `SecurableType.CATALOG`, not `CATALOG`. Both are accepted
- 🚫 **Never enforced.** No privilege is checked anywhere; a passing test here says
nothing about grants in a real workspace

**Status:** ✅ Complete and tested — `tests/unity_catalog/test_grants.py`

---

### 17e. **Workspace Admin** ✅

**Module:** `minilake/services/workspace_admin.py`
**Endpoints:** `/api/2.0/git-credentials`, `/api/2.0/ip-access-lists`,
`/api/2.0/global-init-scripts`, `/api/2.0/notification-destinations`,
`/api/2.0/instance-profiles/*`, `/api/2.0/workspace-conf`

**Key Features:**

- ✅ Real CRUD for all six. Individually dull; together they are what a realistic
`terraform apply` touches before it reaches a catalog, and one 501 fails a whole plan
- ✅ A git credential's `personal_access_token` is stored but **never returned**
- ✅ `list` omits the fields that can carry secrets — the script body for init scripts,
`config` for notification destinations
- ✅ `workspace-conf` values are coerced to strings, because the real API only ever
stores strings; an unset key reads back as `""` rather than being omitted, which is
what lets Terraform see a value it set and later cleared
- 🚫 **Nothing is enforced.** No IP is blocked, no init script runs, no notification is
sent, and an instance profile grants nothing

**Status:** ✅ Complete and tested — `tests/test_workspace_admin.py`

---

### 17f. **Legacy SQL (`preview/sql`)** ✅

**Module:** `minilake/services/preview_sql.py`
**Endpoints:** `/api/2.0/preview/sql/{queries,alerts,dashboards,widgets,visualizations,data_sources}`

**Key Features:**

- ✅ **Adapters, not a second store.** Queries and alerts delegate to `saved_queries.py`
and `alerts.py`, translating wire shapes; an object created through either surface is
visible and editable from the other
- ✅ Translates the flat legacy alert condition (`options.column`/`op`/`value`) to the
modern nested one, including the operator symbol — an untranslated `>` would leave an
alert that silently never fires
- ✅ Correct trash semantics: `DELETE` trashes, `POST .../trash/{id}` **restores**. A
trashed query is hidden from both list surfaces and still retrievable by id
- ✅ The list endpoints are page-numbered. This is not a nicety: the SDK walks them by
incrementing `page` until a response is empty, so a handler that ignores the parameter
never terminates
- ✅ `data_sources` projects the real warehouses, so a legacy client picks an id here and
uses it as `data_source_id`
- 🚫 Dashboards, widgets and visualizations are metadata — nothing renders them

**Status:** ✅ Complete and tested — `tests/test_preview_sql.py`

---

### 18. **MCP Server — LLM agent interface** ✅ (optional)

**Modules:** `minilake/mcp/` (`server.py`, `client.py`, `introspect.py`, `resources.py`,
Expand Down
5 changes: 4 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,13 +99,16 @@ More in [Getting Started](https://github.com/dmux/minilake/blob/main/docs/gettin
| **Clusters** | ✅ Real state machine | CRUD + timed lifecycle transitions, `update`/`pin`/`unpin`; **no real Spark compute** (by design) |
| **Cluster policies & instance pools** | ✅ Real CRUD | Resolve and validate against clusters; never enforced — there is no compute to constrain |
| **Permissions** | ✅ Real CRUD | Single-user "allow-all" default (by design — see Gaps) |
| **UC Grants** | ✅ Real CRUD | `w.grants.*` and `databricks_grants`, with real privilege **inheritance** (a catalog grant is effective on its schemas and tables); never enforced |
| **Workspace admin** | ✅ Real CRUD | Git credentials, IP access lists, global init scripts, notification destinations, instance profiles, `workspace-conf` — stored and read back, never enforced |
| **Legacy SQL (`preview/sql`)** | ✅ Real | Queries, alerts, dashboards, widgets, visualizations, data sources — adapters over the modern stores, so both surfaces see one object |
| **Identity & SCIM** | ✅ Real CRUD | Current user, plus Users/Groups/ServicePrincipals CRUD, SCIM `PATCH` and filtering. Identities are records, not credentials — see Gaps |
| **Tokens** | ✅ Real | `w.tokens.*`; the value is returned once, as in the real API — but authenticates nothing |
| **Persistence** (`MINILAKE_PERSIST=1`) | ✅ Real | JSON snapshot on shutdown, restored on startup |
| **Unity Catalog protocol for Spark** | ✅ Real | `spark.table("cat.sch.tbl")` resolves against minilake — see [Spark & Delta Lake](https://github.com/dmux/minilake/blob/main/docs/spark-and-delta.md#reading-by-name-with-unity-catalog) |
| **JupyterLab + PySpark + Delta** (optional) | ✅ Real | `docker compose --profile notebook up` |
| **MCP Server** (optional, `MINILAKE_MCP=1`) | ✅ Real | 67 tools + resources + prompts at `/mcp` — see [MCP Server](https://github.com/dmux/minilake/blob/main/docs/mcp/index.md) |
| Repos/Git, multi-language notebooks, DBT/pipeline tasks, DLT, Model Registry, Vector Search, Dashboards, Delta Sharing | 🚫 Not implemented | Returns `501 NOT_IMPLEMENTED`. For the measured picture, see [API coverage](https://github.com/dmux/minilake/blob/main/docs/CLI_COVERAGE.md) |
| Repos, multi-language notebooks, DBT/pipeline tasks, DLT, Model Registry, Vector Search, Lakeview/Genie, Delta Sharing | 🚫 Not implemented | Returns `501 NOT_IMPLEMENTED`. For the measured picture, see [API coverage](https://github.com/dmux/minilake/blob/main/docs/CLI_COVERAGE.md) |

## Known Gaps

Expand Down
Loading
Loading
0