8000 Sync main to v0.2.6 by bjoaquinc · Pull Request #19 · dlt-hub/dlthub-init · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,23 @@ jobs:
- name: Build wheel
run: uv build

skills-check:
name: skills drift
runs-on: ubuntu-latest

steps:
- name: Checkout
uses: actions/checkout@v6

- name: Install uv
uses: astral-sh/setup-uv@08807647e7069bb48b6ef5acd8ec9567f424441b # v8.1.0
with:
python-version: "3.13"
enable-cache: true

- name: Verify skills/ matches the pinned workbench ref
run: make check-skills

lock-check:
name: uv.lock drift
runs-on: ubuntu-latest
Expand All @@ -95,3 +112,6 @@ jobs:

- name: Verify bundled workspace lockfile is in sync with its pyproject.toml
run: make scaffold-lock-check

- name: Verify bundled workspace deps match dlt's WORKSPACE_DEPS
run: make scaffold-deps-check
29 changes: 29 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [0.2.10] - 2026-09-15

### Changed
- Bumped the bundled minimal workspace's `dlthub-client` 0.28.3 → 0.28.4, pulling in its new `regex` transitive.

## [0.2.9] - 2026-09-09

### Added
- The bundled minimal workspace now ships a `.mcp.json` registering the dlt workspace MCP server.

### Changed
- Synced the bundled minimal workspace dependencies with dlt's `WORKSPACE_DEPS`: added `s3fs` (plus `aiobotocore`, `aiohttp`, `botocore` and transitives) and dropped the redundant `python_version >= '3.10'` markers on `ibis-framework` and `fastmcp`. Refreshed the scaffold `uv.lock` accordingly.

## [0.2.8] - 2026-08-31

### Changed
- Refreshed the bundled minimal workspace `uv.lock` to the latest within-constraint versions (notably `marimo` 0.23.16 → 0.24.0, `dlthub-client` 0.28.2 → 0.28.3, `authlib` 1.7.2 → 1.8.0, `cyclopts` 4.22.5 → 4.23.3, `cryptography` 50.0.0 → 50.0.1, `pydantic` 2.13.4 → 2.13.5), plus transitive updates. `dlt` and `dlthub` are unchanged.
- Refreshed the root `uv.lock` (`posthog` 7.39.1 → 7.45.1, `pygments` 2.20.0 → 2.21.0, `idna` 3.18 → 3.19).

## [0.2.7] - 2026-08-17

### Changed
- Refreshed the bundled minimal workspace `uv.lock` to the latest within-constraint versions (notably `dlt` 1.29.1 → 1.30.0, `dlthub` 0.29.0 → 0.30.0, `starlette` 1.3.1 → 1.6.0, `sqlglot` 30.14 → 30.17, `setuptools` 83.0 → 84.0), plus transitive updates.

## [0.2.6] - 2026-08-04

### Changed
- Refreshed the bundled minimal workspace `uv.lock` to the latest within-constraint versions (notably `dlt` 1.29.0 → 1.29.1, `duckdb` 1.5.4 → 1.5.5, `sqlglot` 30.13 → 30.14, `cryptography` 49.0 → 50.0, `marimo` 0.23.14 → 0.23.16), plus transitive updates.

## [0.2.5] - 2026-07-22

### Changed
Expand Down
80 changes: 77 additions & 3 deletions CONTRIBUTING.md
< 8000 tr data-hunk="d57a13c9d773c5218abe0ed01318e6caa183c9684fe7ec3c08931194e4221c25" class="show-top-border">
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,25 @@ make ci # everything CI runs
## The bundled workspace

The scaffold's `pyproject.toml` uses loose dependency ranges; the committed
`uv.lock` pins exact versions for reproducible installs. After changing the
scaffold's dependencies, refresh the lock and commit it:
`uv.lock` pins exact versions for reproducible installs.

Its dependency list is generated from dlt's `WORKSPACE_DEPS`, the same list dlt
seeds from its own `dlthub init`. Don't hand-edit it: change `WORKSPACE_DEPS`
upstream, then re-sync, or CI fails and the next sync reverts the edit.

Knobs in `scripts/sync_workspace_deps.py`:
- `BASE_SPECS` — specs this repo owns (`dlt[hub]`, `dlthub`, `dlthub-client`),
emitted ahead of the synced entries and never overwritten by a sync.
- `WORKSPACE_DEPS_MODULE` / `WORKSPACE_DEPS_NAME` — where the list lives in the
dlt wheel; both fail loud if upstream moves or renames it.

The reference dlt version is whatever the scaffold's `uv.lock` resolves, so the
lock is the only pin to maintain.

```bash
make scaffold-lock-upgrade
make scaffold-deps-sync # pull WORKSPACE_DEPS from the locked dlt version
make scaffold-deps-check # CI drift guard
make scaffold-lock-upgrade # refresh the lock after a sync, then commit
```

## Skills
Expand Down Expand Up @@ -92,6 +106,66 @@ For releases, put the real key in a gitignored `.make.env`
(`DLTHUB_INIT_POSTHOG_KEY=phc_…`); the Makefile loads it into `uv build`, and
`make publish` refuses to run without it.

## Testing against a local platform stack

`make workspace-local` scaffolds `./$(WORKSPACE_DIR)` with `api.dlthub.test` and
`auth.dlthub.test` pinned into its `.dlt/config.toml`. `dlthub-init` never talks
to a stack itself, so everything below happens afterwards, against the workspace
that target produces. Workspaces under `examples/` are gitignored and disposable.

`make workspace-dev` pins `api.dlthub.dev` instead. The connect, deploy and
scheduling steps below apply there too, but the TLS and login workarounds are
local-only: the dev stack serves real certificates and shares the auth host with
the API, which is why the target sets neither `DLT_RUNTIME_INSECURE` nor
`AUTH_BASE_URL`. It has not been exercised here.

The local stack serves mkcert certificates that Python does not trust, so every
remote `dlthub` command fails with `CERTIFICATE_VERIFY_FAILED` until one of
these is set:

| Variable | Effect |
|---|---|
| `DLT_RUNTIME_INSECURE` | Skip TLS verification (`1`/`true`/`yes`). This is what `make workspace-local` prints. Covers auth, API, dataplane, uploads and log streaming; every other client stays verified by default. |
| `SSL_CERT_FILE` | Trust the local CA instead: `$(mkcert -CAROOT)/rootCA.pem`. Keeps verification on. |

Log in with the device flow. The default browser loopback flow fails against the
local mock identity service with `invalid_grant`.

```bash
uv run dlthub login --device # prints a verification URL and a resume code
uv run dlthub login --resume <code> # after approving in the browser
```

Then connect and deploy. Non-interactive runs need explicit ids, and `connect`
rebinds the directory, writing `workspace_id` and `organization_id` under
`[runtime]` plus the name under `[workspace.settings]`.

```bash
uv run dlthub workspace list
uv run dlthub workspace connect --create <name> --org-id <org-uuid>
uv run dlthub deploy --dry-run # preview; --show-manifest dumps the YAML
uv run dlthub deploy
```

Jobs come from `__deployment__.py`. Schedule them with cron triggers:

```python
from dlt.hub import run
from dlt.hub.run import trigger

@run.pipeline("hourly_metrics", trigger=trigger.schedule("0 * * * *"))
def load_hourly_metrics():
...
```

Two things that cost time:

- `deploy` registers jobs and their schedules but never runs them. Force a run
with `uv run dlthub run <job>`.
- `destination="warehouse"`, which the bundled skills recommend, is not defined
in the scaffold. Use `destination="duckdb"`, or add the named destination to
`.dlt/config.toml`.

## Code style

Write self-explanatory code. Do not add comments that narrate what the code
Expand Down
10 changes: 8 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
.DEFAULT_GOAL := help

.PHONY: help dev lint lint-fix format format-check fl lint-ci test test-integration build clean-dist version-upgrade version-upgrade-patch version-upgrade-minor version-upgrade-major require-posthog-key publish ci lock-upgrade lock-check scaffold-lock-upgrade scaffold-lock-check generate-skills update-skills check-skills workspace workspace-init workspace-env workspace-local workspace-dev
.PHONY: help dev lint lint-fix format format-check fl lint-ci test test-integration build clean-dist version-upgrade version-upgrade-patch version-upgrade-minor version-upgrade-major require-posthog-key publish ci lock-upgrade lock-check scaffold-lock-upgrade scaffold-lock-check scaffold-deps-sync scaffold-deps-check generate-skills update-skills check-skills workspace workspace-init workspace-env workspace-local workspace-dev

PYTHON_SOURCES := src tests tests_integration scripts
SCAFFOLD_DIR ?= src/dlthub_init/scaffolds/minimal_workspace
Expand Down Expand Up @@ -160,6 +160,12 @@ scaffold-lock-check: ## CI guard: fail if the bundled workspace uv.lock is out o
exit 1; \
fi

scaffold-deps-sync: ## Sync the bundled workspace deps with dlt's WORKSPACE_DEPS; review the diff, re-run scaffold-lock-upgrade, and commit
uv run python scripts/sync_workspace_deps.py

scaffold-deps-check: ## CI guard: fail if the bundled workspace deps drift from dlt's WORKSPACE_DEPS
uv run python scripts/sync_workspace_deps.py --check

#
# Skills (pulled from the dltHub AI workbench)
#
Expand Down Expand Up @@ -188,4 +194,4 @@ check-skills: ## CI guard: fail if skills/ differs from the generated output
exit 1; \
fi

ci: lint-ci test test-integration lock-check scaffold-lock-check check-skills build ## Run all CI checks locally
ci: lint-ci test test-integration lock-check scaffold-lock-check scaffold-deps-check check-skills build ## Run all CI checks locally
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
Scaffold a [dltHub](https://dlthub.com) workspace into a new or existing directory.

`dlthub-init` lays down the files of a minimal dltHub workspace — `pyproject.toml`,
a locked `uv.lock`, and the `.dlt/` configuration — and optionally creates the
virtual environment. That is all it does: it does not log in, run a pipeline, or
a locked `uv.lock`, the `.dlt/` configuration, and a `.mcp.json` registering the
dlt workspace MCP server — and optionally creates the virtual environment. That is all it does: it does not log in, run a pipeline, or
install agent toolkits.

Unlike `dlthub-start`, it is safe to run inside an existing repository. It is
Expand Down Expand Up @@ -41,8 +41,8 @@ install dependencies with `uv`.
- The run stops only if `.dlt/.workspace` already exists — the directory is
already a dltHub workspace (override with `--force`).
- Otherwise existing files are **left alone** and reported as skipped: your
`pyproject.toml`, `.dlt/config.toml`, and `.dlt/secrets.toml` are never
overwritten, and `.gitignore` is skipped (or **merged** with `--merge`).
`pyproject.toml`, `.dlt/config.toml`, `.dlt/secrets.toml`, and `.mcp.json` are
never overwritten, and `.gitignore` is skipped (or **merged** with `--merge`).
- `uv.lock` is written only when `pyproject.toml` is created, so the lockfile
always matches the workspace's dependencies.
- `--force` overwrites the generated files (never secrets).
Expand Down
5 changes: 3 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "hatchling.build"

[project]
name = "dlthub-init"
version = "0.2.5"
version = "0.2.10"
description = "Scaffold a dltHub workspace into a new or existing directory."
readme = "README.md"
requires-python = ">=3.10"
Expand All @@ -30,7 +30,8 @@ Issues = "https://github.com/dlt-hub/dlthub-init/issues"
[project.optional-dependencies]
dev = [
"pytest>=8",
"ruff>=0.6",
# Cap below 0.16: it enables new default lint rules that need a separate migration.
"ruff>=0.6,<0.16",
"mypy>=1.10",
]

Expand Down
Loading
Loading
0