Repository navigation
[BUG] MINILAKE_DUCKDB_MEMORY_LIMIT is never applied to DuckDB #51
Description
Description
The documented memory-limit setting is parsed but never passed to any DuckDB connection.
Reproduced on MiniLake 1.9.0, commit fe8837dcfac9df621dfabc440abc07ebb4cdbcbb, using the repository's Docker test image, a real HTTP server, and fresh disposable state. Image Python sources were verified byte-for-byte against this commit. No mocks or production data were used.
Affected Components
- src/minilake/config.py —
Settings - src/minilake/duckdb_pool.py —
_connect_config - src/minilake/duckdb_pool.py —
_get_uc_connection - src/minilake/duckdb_pool.py —
get_warehouse_connection
Steps to Reproduce
Use a disposable instance and data directory. Paths below are relative to its HTTP base URL; bodies are JSON unless explicitly described as raw bytes. IDs shown as W/J/R/Q/C/HANDLE are replaced with the IDs returned by prior calls.
- Start a fresh instance with
MINILAKE_DUCKDB_MEMORY_LIMIT=16MB. - Create a SQL warehouse.
- Execute
SELECT current_setting('memory_limit') AS configured_limitthrough Statement Execution.
Expected Behavior
The engine reports a 16 MB limit (possibly displayed in equivalent binary units).
Actual Behavior
The reproduced Docker instance reports 6.1 GiB despite the configured 16MB value.
Root Cause Analysis
settings.duckdb_memory_limit is never read outside its declaration. _connect_config only configures extensions, and neither connection creation path sets memory_limit.
Impact
Users cannot constrain engine memory using the advertised setting; queries may consume substantially more than intended and risk memory pressure or process/container termination. No exhaustion test was necessary to establish the ignored setting.
Severity is assessed within MiniLake's documented local development/CI use; it does not assume a shared production Databricks deployment.
Suggested Fix
Apply the configured limit when creating every relevant DuckDB connection, validate invalid limits at startup, and keep shared-connection behavior consistent.
Test Coverage Gaps
Configuration tests check parsed settings but not the actual engine setting. Add a Docker/SDK query of current_setting under a small configured limit.
Severity
Medium
Confidence
High — reproduced through the real HTTP API and traced to the source paths above.
Recommended Labels
bug
Existing Issue Matches
No reasonable duplicate found. Reviewed all open and closed issues and open pull requests in dmux/minilake, then searched component/function/symptom terms for this finding. The only open PR at review time, #29, changes the virtualenv lock entry and does not address this defect. The documented intentional limitations were checked before reporting.
Activity
fix(duckdb_pool): apply MINILAKE_DUCKDB_MEMORY_LIMIT to every DuckDB …