Repository navigation
[BUG] External-table SQL rewriting changes string literals #41
Description
Description
Registering an external table changes the meaning of otherwise unrelated SQL string literals containing its fully qualified name.
Reproduced on MiniLake 1.9.0, commit fe8837dcfac9df621dfabc440abc07ebb4cdbcbb, using the repository's Docker test image, a real HTTP server, and fresh disposable state. Image Python sources were verified byte-for-byte against this commit. No mocks or production data were used.
Affected Components
- src/minilake/services/sql_statements.py —
_rewrite_sql_for_duckdb - src/minilake/services/sql_statements.py —
_execute_sql_real_typed
Steps to Reproduce
Use a disposable instance and data directory. Paths below are relative to its HTTP base URL; bodies are JSON unless explicitly described as raw bytes. IDs shown as W/J/R/Q/C/HANDLE are replaced with the IDs returned by prior calls.
- Create catalog
rewriteand a warehouse W. - POST
/api/2.1/unity-catalog/tableswith{"name":"external","catalog_name":"rewrite","schema_name":"default","table_type":"EXTERNAL","data_source_format":"DELTA","storage_location":"/data/audit_delta"}. No Delta files are needed to reproduce literal rewriting. - Execute
SELECT 'rewrite.default.external' AS valueon W. - Execute
SELECT $$rewrite.default.external$$ AS valueon W.
Expected Behavior
Both statements return the literal string rewrite.default.external unchanged.
Actual Behavior
The single-quoted version fails because rewriting injects nested quotes. The dollar-quoted version succeeds but returns delta_scan('/data/audit_delta') as its value.
Root Cause Analysis
A global regular-expression substitution rewrites all three-part name matches without distinguishing table references from quoted strings, identifiers or comments. The earlier backtick tokenizer does not protect this later pass.
Impact
Queries and writes involving text that happens to name a registered external table fail or silently change values. The same input behaves differently depending on the catalog registry.
Severity is assessed within MiniLake's documented local development/CI use; it does not assume a shared production Databricks deployment.
Suggested Fix
Use a SQL parser or a shared lexical scanner to rewrite only table references in executable SQL, preserving all string/comment/identifier contexts and escaping storage paths.
Test Coverage Gaps
Backtick tests extensively cover strings/comments, but external-table replacement lacks equivalent cases. Cover ordinary, escaped and dollar-quoted strings, comments, and quoted aliases.
Severity
Medium
Confidence
High — reproduced through the real HTTP API and traced to the source paths above.
Recommended Labels
bug
Existing Issue Matches
No reasonable duplicate found. Reviewed all open and closed issues and open pull requests in dmux/minilake, then searched component/function/symptom terms for this finding. The only open PR at review time, #29, changes the virtualenv lock entry and does not address this defect. The documented intentional limitations were checked before reporting.
Activity