Repository navigation
[BUG] Chunked DBFS uploads overwrite files despite overwrite=false #35
Description
Description
Two upload sessions opened for the same absent path with overwrite=false can both close successfully; the later close destroys the earlier upload.
Reproduced on MiniLake 1.9.0, commit fe8837dcfac9df621dfabc440abc07ebb4cdbcbb, using the repository's Docker test image, a real HTTP server, and fresh disposable state. Image Python sources were verified byte-for-byte against this commit. No mocks or production data were used.
Affected Components
Steps to Reproduce
Use a disposable instance and data directory. Paths below are relative to its HTTP base URL; bodies are JSON unless explicitly described as raw bytes. IDs shown as W/J/R/Q/C/HANDLE are replaced with the IDs returned by prior calls.
- POST
/api/2.0/dbfs/createtwice with{"path":"/race","overwrite":false}, retaining handles A and B. - POST
/api/2.0/dbfs/add-blockwith{"handle":A,"data":"Zmlyc3Q="}, then/closewith{"handle":A}. - Add
{"handle":B,"data":"c2Vjb25k"}, then close B. - Read
/raceand base64-decode the result. This interleaving is reproducible sequentially; no timing race is necessary.
Expected Behavior
Closing B fails without replacing the file created by A, or opening B is rejected because the path is already reserved.
Actual Behavior
Both closes return 200; /race contains second, overwriting first.
Root Cause Analysis
The overwrite policy is checked only when opening the handle. close_handle stores but never reads entry["overwrite"], then unconditionally calls write_bytes.
Impact
Interleaved uploads and uploads overlapping a single-shot put can silently lose files despite explicit no-overwrite requests.
Severity is assessed within MiniLake's documented local development/CI use; it does not assume a shared production Databricks deployment.
Suggested Fix
Enforce no-overwrite at final commit using exclusive creation or an atomic no-replace operation. Retain recoverable upload state if commit fails.
Test Coverage Gaps
Chunked-upload tests use one handle with overwrite=true. Add two overlapping handles, an intervening put, and a failed close preserving the existing bytes.
Severity
High
Confidence
High — reproduced through the real HTTP API and traced to the source paths above.
Recommended Labels
bug, high-priority
Existing Issue Matches
No reasonable duplicate found. Reviewed all open and closed issues and open pull requests in dmux/minilake, then searched component/function/symptom terms for this finding. The only open PR at review time, #29, changes the virtualenv lock entry and does not address this defect. The documented intentional limitations were checked before reporting.
Activity
eddiethedean commentedon Oct 25, 2026
Additional scenario now verified through real HTTP on MiniLake 1.9.0 (DuckDB 1.5.6, Python 3.14.3): the competing writer can be the single-shot
putendpoint; two chunked handles are unnecessary.POST /api/2.0/dbfs/createwith{"path":"/late_upload","overwrite":false}; retain handle H.POST /api/2.0/dbfs/putfor/late_uploadwith base64 content encodingcreated after handle; this succeeds because the path is still absent.late uploadto H and close H./late_upload: it containslate upload; the single-shot writer's file has been silently replaced.All steps were sequential, with no timing-dependent race. Add a cross-endpoint regression test and enforce no-replace semantics atomically at handle finalization, preserving both the existing destination and recoverable upload state when close fails.