Repository navigation
[BUG] A SQL task referencing a directory leaves its run permanently RUNNING #44
Description
Description
A valid job request pointing sql_task.file at an existing workspace directory causes an uncaught filesystem exception and never publishes a terminal run status.
Reproduced on MiniLake 1.9.0, commit fe8837dcfac9df621dfabc440abc07ebb4cdbcbb, using the repository's Docker test image, a real HTTP server, and fresh disposable state. Image Python sources were verified byte-for-byte against this commit. No mocks or production data were used.
Affected Components
- src/minilake/services/jobs.py —
_resolve_sql_task_text - src/minilake/services/jobs.py —
_execute_sql_task - src/minilake/services/jobs.py —
_execute_run
Steps to Reproduce
Use a disposable instance and data directory. Paths below are relative to its HTTP base URL; bodies are JSON unless explicitly described as raw bytes. IDs shown as W/J/R/Q/C/HANDLE are replaced with the IDs returned by prior calls.
- Create warehouse W and POST
/api/2.0/workspace/mkdirswith{"path":"/sql_directory"}. - POST
/api/2.2/jobs/runs/submitwith{"tasks":[{"task_key":"bad","sql_task":{"warehouse_id":"W","file":{"path":"/sql_directory"}}}]}(substitute W). - Poll
/api/2.2/jobs/runs/get?run_id=Rand inspect the server log. - Try deleting the run through
/api/2.2/jobs/runs/delete.
Expected Behavior
The task fails with a useful input error and the run reaches a terminal FAILED state; polling and cleanup can complete.
Actual Behavior
After two seconds the tiny run is still RUNNING with no result_state; the background task has already crashed with IsADirectoryError. There is no remaining coroutine to finish it. Active-run deletion is rejected by the service.
Root Cause Analysis
The file resolver checks exists() but not is_file(), then calls read_text(). Only DatabricksError is caught by the task path. _execute_run lacks a failure/finally boundary around gather and terminal-state publication.
Impact
SDK waiters and CI pipelines wait until their own timeouts; the run remains active and cannot be deleted normally. Other unexpected task exceptions take the same unprotected boundary.
Severity is assessed within MiniLake's documented local development/CI use; it does not assume a shared production Databricks deployment.
Suggested Fix
Validate file type and translate decoding/filesystem errors into task failures. Ensure an outer execution boundary always records terminal failure and preserves diagnostic output for unexpected exceptions.
Test Coverage Gaps
SQL-task tests cover missing files and invalid SQL, but not existing directories or invalid text encoding. Add those cases plus unexpected executor failures and assert terminal status.
Severity
Medium
Confidence
High — reproduced through the real HTTP API and traced to the source paths above.
Recommended Labels
bug
Existing Issue Matches
No reasonable duplicate found. Reviewed all open and closed issues and open pull requests in dmux/minilake, then searched component/function/symptom terms for this finding. The only open PR at review time, #29, changes the virtualenv lock entry and does not address this defect. The documented intentional limitations were checked before reporting.
Activity