Repository navigation
[BUG] A delayed start transition resurrects a terminated cluster #46
Description
Description
Terminating a cluster while its start timer is pending first reaches TERMINATED, then silently returns to RUNNING when the old start task fires.
Reproduced on MiniLake 1.9.0, commit fe8837dcfac9df621dfabc440abc07ebb4cdbcbb, using the repository's Docker test image, a real HTTP server, and fresh disposable state. Image Python sources were verified byte-for-byte against this commit. No mocks or production data were used.
Affected Components
- src/minilake/services/clusters.py —
_transition - src/minilake/services/clusters.py —
create_cluster - src/minilake/services/clusters.py —
terminate_cluster
Steps to Reproduce
Use a disposable instance and data directory. Paths below are relative to its HTTP base URL; bodies are JSON unless explicitly described as raw bytes. IDs shown as W/J/R/Q/C/HANDLE are replaced with the IDs returned by prior calls.
- Start with
MINILAKE_CLUSTER_START_DELAY=0.6andMINILAKE_CLUSTER_TERMINATE_DELAY=0.1. - POST
/api/2.1/clusters/createwith{"cluster_name":"race","spark_version":"13.3.x-scala2.12","node_type_id":"Standard_DS3_v2","num_workers":1}; capture C. - Immediately POST
/api/2.1/clusters/deletewith{"cluster_id":"C"}. - GET
/api/2.1/clusters/get?cluster_id=Cafter 0.2 seconds and again after an additional 0.6 seconds.
Expected Behavior
The latest termination request wins; the cluster stays TERMINATED until a later explicit start.
Actual Behavior
First GET reports TERMINATED; the later GET reports RUNNING without any new start request.
Root Cause Analysis
Each lifecycle operation schedules an independent task. _transition checks only that the cluster exists, then writes its target state without checking the operation generation or expected prior state.
Impact
SDK waiters and Terraform observe lifecycle reversals and incorrect state. There is no actual compute here, but the implemented lifecycle emulation becomes unreliable.
Severity is assessed within MiniLake's documented local development/CI use; it does not assume a shared production Databricks deployment.
Suggested Fix
Track/cancel outstanding transitions per cluster or use generation tokens with expected-state checks so stale operations cannot overwrite later requests.
Test Coverage Gaps
Lifecycle tests wait for each operation to finish before starting the next. Add create-then-immediate-terminate and overlapping start/restart/resize/terminate requests.
Severity
Medium
Confidence
High — reproduced through the real HTTP API and traced to the source paths above.
Recommended Labels
bug
Existing Issue Matches
No reasonable duplicate found. Reviewed all open and closed issues and open pull requests in dmux/minilake, then searched component/function/symptom terms for this finding. The only open PR at review time, #29, changes the virtualenv lock entry and does not address this defect. The documented intentional limitations were checked before reporting.
Activity