Repository navigation
chore(scaffold): sync workspace deps with dlt WORKSPACE_DEPS and check for drift - #25
Merged
Merged
Conversation
The bundled scaffold's dependency list is a copy of dlt's WORKSPACE_DEPS, which upstream documents as kept in sync but nothing enforced. The copy sat frozen at dlt's 2026-05-10 state and drifted. scripts/sync_workspace_deps.py reads WORKSPACE_DEPS from the dlt wheel the scaffold's uv.lock resolves, so the lock stays the only pin. It parses the wheel rather than importing dlt, since dlthub-init deliberately does not depend on dlt. 'make scaffold-deps-sync' rewrites the list, and 'make scaffold-deps-check' fails on drift in CI and 'make ci'. BASE_SPECS (dlt[hub], dlthub, dlthub-client) is this repo's own prefix and is never rewritten by a sync; trimming those is a separate decision.
bjoaquinc
force-pushed
the
chore/workspace-deps-sync
branch
from
September 3, 2026 09:19
81a1fdb to
f8b88d9
Compare
Applies the first sync: adds s3fs>=2022.4.0 (dlt-hub/dlt#4044) and drops the python_version >= '3.10' markers on ibis-framework and fastmcp, which upstream removed with Python 3.9 support (dlt-hub/dlt#4074). Re-resolves the bundled uv.lock. s3fs pulls aiobotocore/aiohttp/botocore, taking a synced workspace from 415 MB / 120 packages to 444 MB / 133. Upstream carries it so the job and module launchers can prune it from their runtime requirements and get it pinned from the workspace lock instead of installing it unpinned at deploy time.
bjoaquinc
force-pushed
the
chore/workspace-deps-sync
branch
from
September 3, 2026 09:25
f8b88d9 to
2e01632
Compare
zilto
approved these changes
Sep 3, 2026
There was a problem hiding this comment.
Read all the files, LGTM.
I believe this would benefit from an additional step that checks with uv that the deps written by the script are actually resolvable (probably uv lock --dry-run). Otherwise, it seems that this script could produce an invalid pyproject.toml
(I don't know the full CI/CD cycle, so I might be wrong)
Contributor
Author
|
Already covered: |
Merged
bjoaquinc
added a commit
that referenced
this pull request
Sep 16, 2026
* Refresh uv.lock files (#17) * Refresh uv.lock files Ran uv lock --upgrade on the root project and the minimal_workspace scaffold. Notable scaffold bumps: dlt 1.29.0 -> 1.29.1, duckdb 1.5.4 -> 1.5.5, sqlglot 30.13 -> 30.14, cryptography 49.0 -> 50.0, marimo 0.23.14 -> 0.23.16. Root bumps: posthog, ruff, certifi. Added an [Unreleased] CHANGELOG entry for the scaffold refresh. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Cap ruff below 0.16 to hold the lint ruleset ruff 0.16 turns on new default lint rules that flag pre-existing code; adopting them is a separate migration. Pin ruff <0.16 so the lock refresh does not drag in a new ruleset. Re-locked the root (scaffold lock unaffected). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * Release v0.2.6 (#18) Refreshed the bundled minimal workspace uv.lock (dlt 1.29.0 -> 1.29.1, duckdb 1.5.4 -> 1.5.5, sqlglot 30.13 -> 30.14, cryptography 49.0 -> 50.0, marimo 0.23.14 -> 0.23.16). Capped ruff below 0.16 to hold the lint ruleset. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * Refresh uv.lock files (#20) Ran uv lock --upgrade on the root project and the minimal_workspace scaffold. Notable scaffold bumps: dlt 1.29.1 -> 1.30.0, dlthub 0.29.0 -> 0.30.0, starlette 1.3.1 -> 1.6.0, sqlglot 30.14 -> 30.17, setuptools 83.0 -> 84.0. Root bumps: ast-serialize, librt, mypy, posthog, charset-normalizer, packaging. ruff stays capped below 0.16. Added an [Unreleased] CHANGELOG entry for the scaffold refresh. * Release v0.2.7 (#21) Refreshed the bundled minimal workspace uv.lock (dlt 1.29.1 -> 1.30.0, dlthub 0.29.0 -> 0.30.0, starlette 1.3.1 -> 1.6.0, sqlglot 30.14 -> 30.17, setuptools 83.0 -> 84.0) plus the root dev toolchain lock. * Refresh uv.lock files (#22) Ran uv lock --upgrade on the root project and the minimal_workspace scaffold. Scaffold bumps: marimo 0.23.16 -> 0.24.0, dlthub-client 0.28.2 -> 0.28.3, authlib 1.7.2 -> 1.8.0, cyclopts 4.22.5 -> 4.23.3, cryptography 50.0.0 -> 50.0.1, pydantic 2.13.4 -> 2.13.5, plus transitives; colorama dropped. Root bumps: posthog 7.39.1 -> 7.45.1, pygments 2.20.0 -> 2.21.0, idna 3.18 -> 3.19. dlt and dlthub had no in-constraint bump available. This scaffold ships no marimo notebook, so the 0.24 bump needs no session snapshot lockstep here. Added an [Unreleased] CHANGELOG entry. make ci is green. * Release v0.2.8 (#23) Bump 0.2.7 -> 0.2.8 (pyproject.toml + uv.lock) and promote the [Unreleased] CHANGELOG section to 0.2.8. Ships the lockfile refresh from #22: bundled minimal workspace scaffold picks up marimo 0.23.16 -> 0.24.0, dlthub-client 0.28.2 -> 0.28.3, authlib 1.7.2 -> 1.8.0, cyclopts 4.22.5 -> 4.23.3, cryptography 50.0.0 -> 50.0.1, pydantic 2.13.4 -> 2.13.5, plus transitives. Root dev toolchain picks up posthog 7.39.1 -> 7.45.1, pygments and idna. Patch level: dependency refresh only, no CLI behavior or API change. make ci and dlthub-init --help are green. * Run check-skills in CI (#26) 'make check-skills' was only reachable via 'make ci' locally, so skills/ drifting from the pinned workbench ref would not fail a pull request. Adds a skills-check job mirroring lock-check. Placed above lock-check rather than as another step in it, so this does not collide with the workspace deps drift check landing in a separate PR. * chore(scaffold): sync workspace deps with dlt WORKSPACE_DEPS and check for drift (#25) * Add workspace deps sync and drift check The bundled scaffold's dependency list is a copy of dlt's WORKSPACE_DEPS, which upstream documents as kept in sync but nothing enforced. The copy sat frozen at dlt's 2026-05-10 state and drifted. scripts/sync_workspace_deps.py reads WORKSPACE_DEPS from the dlt wheel the scaffold's uv.lock resolves, so the lock stays the only pin. It parses the wheel rather than importing dlt, since dlthub-init deliberately does not depend on dlt. 'make scaffold-deps-sync' rewrites the list, and 'make scaffold-deps-check' fails on drift in CI and 'make ci'. BASE_SPECS (dlt[hub], dlthub, dlthub-client) is this repo's own prefix and is never rewritten by a sync; trimming those is a separate decision. * Sync scaffold deps with dlt 1.30.0 WORKSPACE_DEPS Applies the first sync: adds s3fs>=2022.4.0 (dlt-hub/dlt#4044) and drops the python_version >= '3.10' markers on ibis-framework and fastmcp, which upstream removed with Python 3.9 support (dlt-hub/dlt#4074). Re-resolves the bundled uv.lock. s3fs pulls aiobotocore/aiohttp/botocore, taking a synced workspace from 415 MB / 120 packages to 444 MB / 133. Upstream carries it so the job and module launchers can prune it from their runtime requirements and get it pinned from the workspace lock instead of installing it unpinned at deploy time. * Document local platform stack testing workflow (#27) * feat(minimal-workspace): ship a .mcp.json registering the dlt workspace MCP server (#28) * feat(minimal-workspace): ship a .mcp.json registering the dlt workspace MCP server * docs(changelog): note the scaffolded .mcp.json * test(scaffold): cover .mcp.json and guard the packaged payload (#29) * Cover scaffold .mcp.json and guard the packaged payload * Document .mcp.json in the README * Release v0.2.9 (#30) * chore(scaffold): bump dlthub-client 0.28.3 to 0.28.4 * Release v0.2.10 (#32) --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Hiba Jamal <35984866+hibajamal@users.noreply.github.com> Co-authored-by: Anton Burnashev <anton.burnashev@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
make scaffold-deps-sync/make scaffold-deps-check(scripts/sync_workspace_deps.py), wired intomake ciand theuv.lock driftCI job.Answers #24: these deps are not chosen here. The list is a copy of dlt's
WORKSPACE_DEPS, which dlt seeds into workspaces from its owndlthub initand documents as kept in sync with ours. Nothing enforced that, so it has sat frozen at dlt's 2026-05-10 state since this repo's initial commit. The reference dlt version now comes from the scaffold's ownuv.lock, soscaffold-lock-upgradestays the only place a version moves.Drift caught
s3fs>=2022.4.0: added upstream in (feat) prunes deps from launcher groups if present in user deps dlt#4044, never came acrossibis-framework/fastmcp: still carry; python_version >= '3.10', dropped upstream in Remove support for python 3.9 dlt#4074 with Python 3.9 supportApplied in the second commit, scaffold
uv.lockre-resolved.Worth a second look
Syncing adds
s3fs, one more dependency to a list #24 already finds too long. Upstream carries it so the job and module launchers prune it and take it pinned from the workspace lock, instead of installing it unpinned at deploy time. Separate commit so it can be dropped, thoughscaffold-deps-checkthen goes red until upstream trims.Trimming belongs in
WORKSPACE_DEPSso dlt's owndlthub initgets the same change, then re-sync here.Verification
make cigreen: lint-ci, 125 unit tests, integration tests (realuv syncof the synced scaffold),lock-check,scaffold-lock-check,scaffold-deps-check,check-skills, build.