Repository navigation
feat(notebooks): Delta OPTIMIZE/VACUUM maintenance notebook - #10
Merged
Merged
Conversation
Seeded alongside the quickstart: fifty micro-batch appends manufacture a small-files problem (200 tiny Parquet files), OPTIMIZE compacts them to four, and VACUUM reclaims the 201 tombstoned ones. Every measurement runs through the SQL Statement Execution API -- glob() for what is on disk, a SQL replay of the _delta_log for what is live -- which is what makes the "files on disk go up after OPTIMIZE" step visible from the Databricks side. Spark work goes through the Jobs API, like the quickstart: the embedded kernel has no pyspark on purpose, so an in-kernel SparkSession is not an option here. Dockerfile.notebook now copies all of notebooks/ rather than one filename. Claude-Session: https://claude.ai/code/session_01Xme3rgBDt5Ce4egA77skms
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a second seeded notebook to the embedded JupyterLab, covering the operational half of Delta Lake that the quickstart does not: the small-files problem and the
OPTIMIZE/VACUUMcycle that fixes it.What it does
Fifty micro-batch appends manufacture the problem — 200 tiny Parquet files, 50 rows each — then
OPTIMIZEcompacts them andVACUUMreclaims the tombstones. Verified end to end:OPTIMIZEVACUUMThe pivot is the on-disk count rising to 204 after
OPTIMIZE: 200 tombstoned files that still back time travel, whichVACUUMthen reclaims — ending withFileNotFoundExceptionon version 0. Also covered:DESCRIBE HISTORY,versionAsOf,OPTIMIZE ... WHERE ... ZORDER BY, and aVACUUM ... DRY RUN.Both sides of the boundary
Every measurement runs through the SQL Statement Execution API with
databricks-sdk—glob()for what is on disk, and a SQL replay of the_delta_log(addpaths minusremovepaths) for what is live. That replay is what makes the tombstone step visible from the Databricks side at all.Spark work goes through the Jobs API, like the quickstart: the embedded kernel has no
pysparkon purpose, so an in-kernelSparkSessionis not an option.OPTIMIZE,VACUUM,DESCRIBE HISTORYandVERSION AS OFare Delta protocol operations and are not in minilake's DuckDB dialect — section 11 of the notebook states that outright rather than papering over it.Also here
Dockerfile.notebookcopies all ofnotebooks/rather than one hardcoded filename.docs/spark-and-delta.md— the JupyterLab section now describes the embedded Jupyter (both seeded notebooks, and the deliberate absence ofpyspark) before the optional--profile notebookcontainer.FEATURES.md§12a lists both seeded notebooks; §12 goes back to describing only the local-Spark container.Verification
Executed non-interactively (
jupyter nbconvert --execute) againstghcr.io/dmux/minilake:latestin an isolated container and volume: ~90s, zero errors, shipped with outputs cleared.https://claude.ai/code/session_01Xme3rgBDt5Ce4egA77skms