Repository navigation
[BUG] jobs/run-now ignores idempotency_token and executes duplicate writes on retry #56
Description
Description
Repeating a jobs/run-now request with the same idempotency_token starts another run and executes its tasks again. A normal client retry can therefore duplicate data even when the caller explicitly requested idempotent execution.
Verified against MiniLake 1.9.0 source at local commit e24f46a7753ff0be30df3d83978c370c817f89cd (based on upstream fe8837dcfac9df621dfabc440abc07ebb4cdbcbb; the affected code is unchanged). Reproduced using a real disposable HTTP server and fresh data, with Python 3.14.3, DuckDB 1.5.6, FastAPI 0.162.0 and, for MCP, mcp 1.30.0. No mocks or user data were used.
Affected Components
- models/jobs.py:224 —
RunNowRequest - services/jobs.py:612 —
run_now POST /api/2.2/jobs/run-now
Steps to Reproduce
Use a fresh disposable instance; paths are relative to its base URL.
- POST
/api/2.0/sql/warehouseswith{"name":"retry-repro"}; capture W. - Execute
CREATE TABLE retry_effect (id INTEGER)viaPOST /api/2.0/sql/statementswith W. - POST raw bytes
INSERT INTO retry_effect VALUES (99)to/api/2.0/workspace-files/import-file/retry.sql. - POST
/api/2.2/jobs/createwith:Capture job ID J; substitute the actual W.{"name":"retry-repro","tasks":[{"task_key":"insert","sql_task":{"warehouse_id":"W","file":{"path":"/retry.sql"}}}]} - POST
/api/2.2/jobs/run-nowwith{"job_id":J,"idempotency_token":"one-logical-write"}. Poll/api/2.2/jobs/runs/get?run_id=Runtil TERMINATED. - Repeat the identical request from step 5 and wait for termination again.
- Query
SELECT * FROM retry_effect.
Expected Behavior
The second request returns the original run ID and does not re-execute the INSERT. The table contains one row.
This matches the supported Databricks SDK's JobsAPI.run_now(idempotency_token=...) contract: a repeated token returns the existing run rather than launching a second run.
Actual Behavior
The reproduction returned run IDs 4 and 6, both completed successfully, and the table contained [[99],[99]].
Root Cause Analysis
RunNowRequest accepts extra fields, including the token, but run_now never reads it. Every call allocates a fresh run ID and launches _execute_run. The separate runs/submit handler implements a token lookup; run-now does not.
Impact
Retry-safe client code behaves incorrectly against the emulator. Retried ingestion, append, or other tasks repeat their side effects and corrupt fixtures/datasets. Tests can fail to catch broken idempotency assumptions before deployment. The defect requires no concurrency or timing race: sequential identical calls reproduce it.
Suggested Fix
Implement explicit token validation and token-to-run tracking for run-now, returning the original run for repeats. Preserve the mapping across persistence and define deleted-run behavior consistently with the SDK contract. Ensure check-and-create is atomic within the supported process model.
Test Coverage Gaps
tests/test_jobs_submit.py::test_submit_idempotency_token_reuses_the_run covers only runs/submit. Add a run-now counterpart using a real side-effecting SQL task; assert identical returned IDs, one stored run and one inserted row after retries, including after restart.
Severity
High
Confidence
High — reproduced sequentially through real HTTP and confirmed duplicate stored rows.
Recommended Labels
bug, high-priority
Existing Issue Matches
No reasonable duplicate found. Reviewed all open issues, the closed-issue search (no results), and all open PRs, then searched the affected functions/components and symptoms. Open PR #54 fixes duplicate Arrow columns only; #29 updates virtualenv. Neither addresses this defect. The documented local/single-user limitations were checked. Searches included run_now, idempotency, retry and duplicate execution.
Activity