Repository navigation
[BUG] Statement cache overflow erases the newest result before its download #33
Description
Description
Crossing the cache limit clears results from every retained statement, including the statement just executed. A successful EXTERNAL_LINKS response can immediately download an empty array.
Reproduced on MiniLake 1.9.0, commit fe8837dcfac9df621dfabc440abc07ebb4cdbcbb, using the repository's Docker test image, a real HTTP server, and fresh disposable state. Image Python sources were verified byte-for-byte against this commit. No mocks or production data were used.
Affected Components
- src/minilake/services/sql_statements.py —
_record_statement - src/minilake/services/sql_statements.py —
get_result_chunk_data
Steps to Reproduce
Use a disposable instance and data directory. Paths below are relative to its HTTP base URL; bodies are JSON unless explicitly described as raw bytes. IDs shown as W/J/R/Q/C/HANDLE are replaced with the IDs returned by prior calls.
- Start with
MINILAKE_STATEMENT_CACHE_SIZE=2and fresh state. Create a SQL warehouse. - POST three statements to
/api/2.0/sql/statements, each withwarehouse_id,disposition:"EXTERNAL_LINKS",format:"JSON_ARRAY", and respectivelySELECT 1 AS n,SELECT 2 AS n,SELECT 3 AS n. - Immediately GET the third response's
result.external_links[0].external_link. - GET
/api/2.0/sql/statements/{third_statement_id}.
Expected Behavior
Only the oldest excess entry is evicted. The newest external link yields [[3]] and its manifest continues to report one row.
Actual Behavior
The POST advertises one row but the external link returns []. Fetching the newest statement reports zero rows and result:null. The default limit of 500 reaches the same branch at statement 501.
Root Cause Analysis
Clearing raw_rows and result does not reduce len(statements), so the first loop continues across all entries. Only afterward does the while loop remove the excess metadata entries.
Impact
Long-running instances silently return missing query results after a routine number of statements. Consumers can interpret the empty download as real empty business data.
Severity is assessed within MiniLake's documented local development/CI use; it does not assume a shared production Databricks deployment.
Suggested Fix
Choose eviction victims before clearing payloads, evict only as many entries as required, and preserve the newly inserted result. If metadata and payload budgets differ, track them separately. Downloads for evicted payloads should return an explicit unavailable result.
Test Coverage Gaps
tests/test_query_history.py tests pagination with a handful of statements, but never crosses the configured cache limit and downloads the newest external result. Add limit, limit+1 and repeated-overflow tests.
Severity
High
Confidence
High — reproduced through the real HTTP API and traced to the source paths above.
Recommended Labels
bug, high-priority
Existing Issue Matches
No reasonable duplicate found. Reviewed all open and closed issues and open pull requests in dmux/minilake, then searched component/function/symptom terms for this finding. The only open PR at review time, #29, changes the virtualenv lock entry and does not address this defect. The documented intentional limitations were checked before reporting.
Activity