8000 feat: embedded web UI and JupyterLab (1.7.0) by dmux · Pull Request #4 · dmux/minilake · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,13 @@ Thumbs.db
docs/
examples/
CONTRIBUTING.md
# Frontend build artifacts. `COPY ui/ ./` in the frontend-builder stage would
# otherwise drop the host's node_modules over the ones just installed in the
# image — a different platform's binaries, and a much larger build context.
ui/node_modules/
ui/.next/
ui/out/
# Monaco is copied in from node_modules by the ui build itself.
ui/public/monaco/
# Staged by the ui build on the host; the image copies it from frontend-builder.
src/minilake/ui_static/
29 changes: 29 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,35 @@ jobs:
- name: Run pre-commit hooks (ruff lint + format, hygiene checks)
uses: pre-commit/action@v3.0.1

ui:
runs-on: ubuntu-latest
defaults:
run:
working-directory: ui
steps:
- uses: actions/checkout@v4

- uses: actions/setup-node@v4
with:
node-version: "24"

- name: Install dependencies
run: |
corepack enable
pnpm install --frozen-lockfile

- name: Lint
run: pnpm lint

- name: Typecheck
run: pnpm typecheck

- name: Unit tests
run: pnpm test

- name: Build static export
run: pnpm build

test:
runs-on: ubuntu-latest
steps:
Expand Down
13 changes: 13 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,19 @@ jobs:
steps:
- uses: actions/checkout@v4

# The wheel force-includes ui/out, which is generated, not committed. Without
# this step `uv build` silently produces a wheel with no web UI.
- uses: actions/setup-node@v4
with:
node-version: "24"

- name: Build the web UI
run: |
corepack enable
pnpm install --frozen-lockfile
pnpm run build
working-directory: ui

- name: Set up uv
uses: astral-sh/setup-uv@v5

Expand Down
34 changes: 22 additions & 12 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,22 @@ __pycache__/
*$py.class
*.so
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
# Anchored to the repo root on purpose. Unanchored, `lib/` also matches
# `ui/src/lib/`, which silently kept the web UI's entire API client and helpers
# out of git — the build then worked locally, from files on disk, and failed in
# CI from a clean checkout. The same trap applies to `build/`, `dist/` and `var/`.
/build/
/develop-eggs/
/dist/
/downloads/
/eggs/
/.eggs/
/lib/
/lib64/
/parts/
/sdist/
/var/
/wheels/
*.egg-info/
.installed.cfg
*.egg
Expand Down Expand Up @@ -69,3 +73,9 @@ Thumbs.db
# Claude Code local settings (machine-specific; .claude/CLAUDE.md IS tracked)
.claude/settings.local.json
.claude/scheduled_tasks.lock

# Built web UI, staged into the package by `cd ui && pnpm build` (see docs/ui.md)
src/minilake/ui_static/

# Local agent tooling, alongside .claude/ and .mcp.json — not project configuration
.agents/
113 changes: 113 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,113 @@
# Changelog

All notable changes to minilake are recorded here.

The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the
project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

Per-feature design rationale and known limitations live in [FEATURES.md](FEATURES.md);
this file records what changed between releases.

## [1.7.0] — 2026-08-18

The release that gives minilake a face: an embedded web workspace, and a real
JupyterLab inside the image.

### Added

**Web UI** — an Athena-style SQL workspace at `/ui`, served by minilake itself as a
Next.js static export. Same origin as the API, no second process, no extra port.

- Query editor with multiple tabs, a Monaco editor whose SQL completion is fed by live
Unity Catalog metadata, run/cancel/format/`EXPLAIN`, and an auto-limit that wraps a
bare `SELECT` while leaving explicit `LIMIT`s and DDL alone.
- Virtualized result grid with column types, sorting, filtering, CSV/JSON download, a
chart tab and query statistics.
- Data catalog tree down to columns, with table details, *Preview table* and
*Generate table DDL* — plus create and drop for catalogs, schemas, MANAGED tables
and volumes. Drops cascade on the server, and the confirmation says so.
- Jobs: list and run, and now create, edit and delete, with a per-task run detail view
alongside the run logs. The editor offers only the task types minilake actually
executes (`notebook_task`, `spark_python_task`, `sql_task.file`), since the others
are reported `SKIPPED` rather than faked.
- Workspace browser over `/Workspace` — the tree `databricks bundle deploy` and the
Jobs API write into, with a read-only content viewer.
- Clusters driving the real state machine, and secret scopes and keys. Secret values
are never displayed: the API has no read path for them, by design.
- Warehouses, files, saved queries, recent queries, and a settings page with server
health, the signed-in identity and state reset.
- Light, dark and system themes. Monaco is bundled rather than fetched from a CDN, so
the image stays offline-capable.

**Embedded JupyterLab** — a real notebook server running inside minilake, reverse-proxied
at `/jupyter` and framed by the UI at `/ui/notebooks`. On by default
(`MINILAKE_NOTEBOOK=0` disables it).

- Serving it same-origin is what makes the embedded page possible at all: jupyter-server
sends `Content-Security-Policy: frame-ancestors 'self'`, so a notebook server on its
own port cannot be framed by the UI. Proxying it satisfies `'self'` without weakening
the policy.
- The proxy relays WebSockets, so kernels connect, execute and stream output normally;
HTTP responses stream rather than buffer, for JupyterLab's multi-megabyte bundles.
- The Jupyter process binds `127.0.0.1` and is reachable only through minilake, so it is
exposed exactly as far as minilake is and no further.
- **No second Spark in the image.** Notebooks reach real Spark through the Jobs API, in
the same sibling containers a job uses. The bundled quickstart carries the helper that
stages a script, runs it and returns its output.
- A quickstart notebook is seeded into the notebook directory on first start and never
overwritten afterwards, so edits survive an upgrade.
- New optional extra `minilake[notebook]`, bundled in the Docker image.
- New settings: `MINILAKE_NOTEBOOK`, `MINILAKE_NOTEBOOK_PATH`, `MINILAKE_NOTEBOOK_PORT`,
`MINILAKE_NOTEBOOK_DIR`.

**Query History API** — `GET /api/2.0/sql/history/queries`, backed by the statement cache
rather than a second store, so the two cannot drift. Records **failed** statements, which
is usually why the panel gets opened. Supports `filter_by.statuses` /
`.warehouse_ids` / `.statement_ids`, `max_results` with an offset `page_token`, and
`include_metrics` for the metrics minilake can actually measure.

**Saved Queries API** — full CRUD at `/api/2.0/sql/queries`, honouring `update_mask` and
`auto_resolve_display_name`, and persisted with `MINILAKE_PERSIST=1`.

### Changed

- `POST /_minilake/reset?full=true` now honours `full`. It previously accepted the
parameter and ignored it, so the UI's "Full reset" and "Reset state" buttons did the
same thing. A full reset deletes the on-disk data directories — the per-catalog DuckDB
databases, volume directories and warehouse files that no service `reset()` touches —
while a plain reset still only clears the in-memory registries.
- Deleting a SQL warehouse from the UI now asks for confirmation, matching every other
destructive action.

### Fixed

- Font MIME types and CSS variable loading for the embedded UI: on a bare `python:slim`
image there is no `/etc/mime.types`, so Python guessed `application/octet-stream` for
hashed `.woff2` assets and the browser refused to apply the font.
- React hydration error #418, and an invalid CSS import.

### Removed

- `ConfigItem` and `ConfigUpdateRequest` from `admin.py`. They modelled a
`/_minilake/config` endpoint that does not exist.

### Documentation

- New [docs/ui.md](docs/ui.md) covering the web UI, how it is built and how it is served.
- `FEATURES.md` no longer lists `query_history` and `queries` among the unimplemented
501 modules — both are implemented and in use by the UI.
- `permissions` and `dbfs` are now recorded as deliberate UI omissions: permissions are
stored but never enforced, and DBFS shares its on-disk root with the Files API, which
the Files page already browses.

## [1.6.0] — 2026-08-08

Baseline for this changelog. Core SQL and Unity Catalog with per-catalog isolation, Jobs
with real DAG scheduling and Spark execution in sibling containers, Workspace, DBFS,
Files, Secrets, Clusters, Permissions, real Spark/Delta round-trips, and the optional MCP
server — all working and tested, with `MINILAKE_PERSIST` wired in.

See [FEATURES.md](FEATURES.md) for the full per-feature status of this release.

[1.7.0]: https://github.com/dmux/minilake/compare/v1.6.0...v1.7.0
[1.6.0]: https://github.com/dmux/minilake/releases/tag/v1.6.0
29 changes: 24 additions & 5 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,3 +1,14 @@
# Build the embedded web UI (Next.js static export). Its `prebuild` step also copies
# Monaco out of node_modules into public/, so the SQL editor is served by minilake
# rather than fetched from a CDN at runtime — the image is meant to work offline.
FROM node:24-slim AS frontend-builder
WORKDIR /app/ui
RUN corepack enable
COPY ui/package.json ui/pnpm-lock.yaml ui/pnpm-workspace.yaml ./
RUN pnpm install --frozen-lockfile
COPY ui/ ./
RUN pnpm run build

# Resolve the Delta and Unity Catalog connector jars at build time, using the same Ivy
# resolver spark-submit uses at runtime, so a job container never needs Maven Central.
# Reusing spark-submit (rather than curl-ing jars by hand) is deliberate: it writes the
Expand All @@ -23,7 +34,6 @@ RUN mkdir -p /opt/ivy/cache /opt/ivy/jars && \
/tmp/warmup.py && \
ls /opt/ivy/jars/*.jar > /dev/null


FROM python:3.11-slim

WORKDIR /opt/minilake
Expand All @@ -38,10 +48,19 @@ RUN apt-get update && apt-get install -y \
COPY pyproject.toml pyproject.toml
COPY src/ src/
COPY README.md README.md

# Install minilake with the MCP extra. Baked into the image so MINILAKE_MCP=1 is all that's
# needed to turn the MCP server on; the extra stays optional for PyPI installs.
RUN pip install --no-cache-dir -e ".[mcp]"
# Inside the package, not beside it: `app.py` looks for `minilake/ui_static` first,
# which is also where the wheel's force-include puts it — so the container and a
# `pip install minilake` resolve the UI the same way.
COPY --from=frontend-builder /app/ui/out /opt/minilake/src/minilake/ui_static

# Install minilake with the MCP and notebook extras. Both are baked into the image so
# MINILAKE_MCP=1 is all that's needed to turn the MCP server on and JupyterLab is there
# by default; the extras stay optional for PyPI installs.
#
# The notebook extra brings JupyterLab but deliberately no pyspark: notebooks reach real
# Spark through the Jobs API, in the same sibling containers a job uses, so the image
# never carries a second Spark.
RUN pip install --no-cache-dir -e ".[mcp,notebook]"

# Pre-install DuckDB's `delta` extension. Without this the server downloads it from
# extensions.duckdb.org on every first boot, into a $HOME path that is not a volume — so it
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile.test
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ COPY tests/ tests/
COPY README.md README.md

# Install minilake + all dev dependencies
RUN uv pip install -e ".[mcp]" --python /usr/local/bin/python && \
RUN uv pip install -e ".[mcp,notebook]" --python /usr/local/bin/python && \
uv pip install \
pytest>=8.0 \
pytest-xdist>=3.6 \
Expand Down
Loading
Loading
0