Repository navigation
[BUG] Files API stores volume content outside the Unity Catalog volume directory #38
Description
Description
Files uploaded under /Volumes/catalog/schema/volume/... are written under the generic DBFS tree, whereas UC volume create/delete operate on a different directory. Deleting a managed volume leaves its API-uploaded files accessible.
Reproduced on MiniLake 1.9.0, commit fe8837dcfac9df621dfabc440abc07ebb4cdbcbb, using the repository's Docker test image, a real HTTP server, and fresh disposable state. Image Python sources were verified byte-for-byte against this commit. No mocks or production data were used.
Affected Components
- src/minilake/services/files.py —
_resolve - src/minilake/services/dbfs.py —
_resolve - src/minilake/services/unity_catalog.py —
_get_volumes_dir - src/minilake/services/unity_catalog.py —
delete_volume
Steps to Reproduce
Use a disposable instance and data directory. Paths below are relative to its HTTP base URL; bodies are JSON unless explicitly described as raw bytes. IDs shown as W/J/R/Q/C/HANDLE are replaced with the IDs returned by prior calls.
- Create catalog
namespace, then POST/api/2.1/unity-catalog/volumeswith{"catalog_name":"namespace","schema_name":"default","name":"v","volume_type":"MANAGED"}. - PUT raw bytes
old volume bytesto/api/2.0/fs/files/Volumes/namespace/default/v/payload.txt. - DELETE
/api/2.1/unity-catalog/volumes/namespace.default.v. - GET the exact Files API path from step 2; optionally recreate the same UC volume and read it again.
- Inspect the data directory: bytes reside at
dbfs/Volumes/namespace/default/v/payload.txt, while UC usesvolumes/namespace/default/v.
Expected Behavior
Volume URIs resolve to the actual registered volume storage, and deletion of that managed volume makes its content unavailable through Files/DBFS.
Actual Behavior
Volume deletion succeeds, but GET still returns HTTP 200 and old volume bytes. The actual UC volume directory is removed while the uploaded bytes remain elsewhere.
Root Cause Analysis
Files and DBFS treat the Volumes prefix as a literal subdirectory of data_dir/dbfs; Unity Catalog manages data_dir/volumes. No shared resolver connects the namespaces.
Impact
Cleanup reports success while content survives and reappears under recreated names. Jobs/consumers using the actual volume directory see different files from API users.
Severity is assessed within MiniLake's documented local development/CI use; it does not assume a shared production Databricks deployment.
Suggested Fix
Introduce one volume-aware resolver shared by Files, DBFS and Unity Catalog. Resolve registered MANAGED/EXTERNAL volume storage consistently and enforce lifecycle checks.
Test Coverage Gaps
Files tests and volume CRUD tests run separately. Add create-volume/upload/read/delete/recreate flows, plus cross-API visibility and nonexistent-volume paths.
Severity
High
Confidence
High — reproduced through the real HTTP API and traced to the source paths above.
Recommended Labels
bug, high-priority
Existing Issue Matches
No reasonable duplicate found. Reviewed all open and closed issues and open pull requests in dmux/minilake, then searched component/function/symptom terms for this finding. The only open PR at review time, #29, changes the virtualenv lock entry and does not address this defect. The documented intentional limitations were checked before reporting.
API Reference
Files API documentation defines /Volumes/<catalog>/<schema>/<volume>/<path> as a volume URI.
Activity