Repository navigation
[BUG] An unfinished DBFS upload corrupts the entire persistence snapshot #31
Description
Description
A graceful shutdown with any open DBFS upload handle destroys the JSON snapshot, including unrelated catalogs, jobs, secrets and saved queries. A previously valid snapshot is overwritten with partial JSON.
Reproduced on MiniLake 1.9.0, commit fe8837dcfac9df621dfabc440abc07ebb4cdbcbb, using the repository's Docker test image, a real HTTP server, and fresh disposable state. Image Python sources were verified byte-for-byte against this commit. No mocks or production data were used.
Affected Components
- src/minilake/services/dbfs.py —
create_handle - src/minilake/services/dbfs.py —
get_state - src/minilake/persistence.py —
save_state - src/minilake/persistence.py —
load_state
Steps to Reproduce
Use a disposable instance and data directory. Paths below are relative to its HTTP base URL; bodies are JSON unless explicitly described as raw bytes. IDs shown as W/J/R/Q/C/HANDLE are replaced with the IDs returned by prior calls.
- Start with
MINILAKE_PERSIST=1and a fresh absoluteMINILAKE_DATA_DIR. POST /api/2.1/unity-catalog/catalogswith{"name":"snapshot_victim"}. Gracefully stop and restart using the same directory; GET that catalog to establish a valid existing snapshot.POST /api/2.0/dbfs/createwith{"path":"/unfinished","overwrite":false}. Capturehandle.POST /api/2.0/dbfs/add-blockwith{"handle":HANDLE,"data":"cGVuZGluZw=="}. Leave the handle open.- Gracefully stop and restart again with the same directory, then
GET /api/2.1/unity-catalog/catalogs/snapshot_victim.
Expected Behavior
The catalog and other persistent state survive. Ephemeral upload sessions are omitted or safely serialized, and a failed save preserves the previous valid snapshot.
Actual Behavior
GET returns 404. Shutdown logs Object of type bytearray is not JSON serializable; startup logs Failed to load state snapshot: Expecting value. The whole envelope is unreadable.
Root Cause Analysis
create_handle stores a bytearray; get_state exposes it. save_state opens the destination with mode w before json.dump, truncating the prior snapshot. Serialization errors are caught after partial output has been written.
Impact
Any interrupted client upload followed by an ordinary restart can make all service metadata disappear. Physical database files can remain, but registrations and other JSON-only state are unavailable. This is the most dangerous finding because one ephemeral upload affects the entire workspace.
Severity is assessed within MiniLake's documented local development/CI use; it does not assume a shared production Databricks deployment.
Suggested Fix
Exclude ephemeral DBFS handles from snapshots. Serialize and validate before replacing the destination, write a temporary file in the same directory, flush it, and replace atomically. Preserve/report save failures without destroying the last good snapshot.
Test Coverage Gaps
tests/test_persistence.py restarts with a catalog only; tests/test_dbfs.py always closes uploads. Add a process-restart test with a valid previous snapshot plus an open upload, and a failed-serialization test asserting the previous snapshot remains readable.
Severity
High
Confidence
High — reproduced through the real HTTP API and traced to the source paths above.
Recommended Labels
bug, high-priority
Existing Issue Matches
No reasonable duplicate found. Reviewed all open and closed issues and open pull requests in dmux/minilake, then searched component/function/symptom terms for this finding. The only open PR at review time, #29, changes the virtualenv lock entry and does not address this defect. The documented intentional limitations were checked before reporting.
Activity
eddiethedean commentedon Oct 25, 2026
Additional minimal reproduction verified on MiniLake 1.9.0 (DuckDB 1.5.6, Python 3.14.3), using real HTTP and a fresh disposable data directory: no
add-blockcall is needed. Merely opening a DBFS upload handle and leaving it unclosed corrupts persistence on graceful shutdown, because the initial emptybytearray()is already non-serializable.I first established a valid 2,532-byte snapshot containing catalog
snapshot_guard, restarted successfully, called onlyPOST /api/2.0/dbfs/createwith{"path":"/unfinished"}, then shut down. The previous snapshot was replaced by 777 bytes of incomplete JSON (Expecting value: line 35 column 21 (char 777)). After restart, catalog listing was empty.Add a zero-block-upload restart case alongside the nonempty-buffer case, and assert a failed save leaves the previous snapshot intact. This confirms even an abandoned empty upload can destroy unrelated persisted metadata.