Repository navigation
[BUG] Persisted jobs and runs become inaccessible after restart #36
Description
Description
With persistence enabled, jobs remain visible in list responses after restart but get/run/update operations return NOT_FOUND for their advertised numeric IDs.
Reproduced on MiniLake 1.9.0, commit fe8837dcfac9df621dfabc440abc07ebb4cdbcbb, using the repository's Docker test image, a real HTTP server, and fresh disposable state. Image Python sources were verified byte-for-byte against this commit. No mocks or production data were used.
Affected Components
- src/minilake/services/jobs.py —
restore_state - src/minilake/services/jobs.py —
get_job - src/minilake/services/jobs.py —
get_run - src/minilake/services/jobs.py —
get_run_output
Steps to Reproduce
Use a disposable instance and data directory. Paths below are relative to its HTTP base URL; bodies are JSON unless explicitly described as raw bytes. IDs shown as W/J/R/Q/C/HANDLE are replaced with the IDs returned by prior calls.
- Start with
MINILAKE_PERSIST=1and a fresh absolute data directory. - POST
/api/2.2/jobs/createwith{"name":"persisted"}; capture numeric job ID J. - POST
/api/2.2/jobs/run-nowwith{"job_id":J}; capture run ID R and wait for TERMINATED. - Gracefully stop/restart against the same directory.
- GET
/api/2.2/jobs/list,/api/2.2/jobs/get?job_id=J, and/api/2.2/jobs/runs/get?run_id=R.
Expected Behavior
IDs returned before restart and in listings continue to address the same jobs and completed runs.
Actual Behavior
The list includes job 1, but get-job and get-run both return 404 (Job 1 not found / Run 1 not found).
Root Cause Analysis
JSON converts integer dictionary keys to strings. restore_state blindly updates jobs/runs/task_outputs; API lookups still use integers. The same conversion affects persisted task-output lookup.
Impact
Every persisted job definition becomes unusable through normal ID-based operations after restart. Completed run history cannot be retrieved by ID, undermining the documented persistence guarantee.
Severity is assessed within MiniLake's documented local development/CI use; it does not assume a shared production Databricks deployment.
Suggested Fix
Normalize numeric-keyed registries during restore or use a consistent string key representation throughout the service. Verify submit-token mappings still resolve to restored runs.
Test Coverage Gaps
The only process-restart persistence test creates a catalog, not jobs/runs. Add a completed SQL job, restart, and exercise list/get/run-now/update/output with the original IDs.
Severity
High
Confidence
High — reproduced through the real HTTP API and traced to the source paths above.
Recommended Labels
bug, high-priority
Existing Issue Matches
No reasonable duplicate found. Reviewed all open and closed issues and open pull requests in dmux/minilake, then searched component/function/symptom terms for this finding. The only open PR at review time, #29, changes the virtualenv lock entry and does not address this defect. The documented intentional limitations were checked before reporting.
Activity