Repository navigation
[BUG] Arrow results silently discard columns with duplicate names #37
Description
Description
ARROW_STREAM builds a table through a name-keyed dictionary, dropping all but the last result column for each repeated name.
Reproduced on MiniLake 1.9.0, commit fe8837dcfac9df621dfabc440abc07ebb4cdbcbb, using the repository's Docker test image, a real HTTP server, and fresh disposable state. Image Python sources were verified byte-for-byte against this commit. No mocks or production data were used.
Affected Components
Steps to Reproduce
Use a disposable instance and data directory. Paths below are relative to its HTTP base URL; bodies are JSON unless explicitly described as raw bytes. IDs shown as W/J/R/Q/C/HANDLE are replaced with the IDs returned by prior calls.
- Create a SQL warehouse.
- POST
/api/2.0/sql/statementswith its ID,statement:"SELECT 1 AS same, 2 AS same",disposition:"EXTERNAL_LINKS",format:"ARROW_STREAM". - Download
result.external_links[0].external_link. - Read with
pyarrow.ipc.open_stream(downloaded_bytes).read_all(); compare its schema/row tomanifest.schema.
Expected Behavior
Two columns named same, in original order, containing 1 and 2. Arrow permits repeated field names.
Actual Behavior
The manifest advertises both columns, while the actual Arrow table contains only one same column with value 2.
Root Cause Analysis
pa.table({col: arrays[i] for i, col in enumerate(columns)}) collapses repeated names before Arrow sees them.
Impact
Valid SQL joins and repeated aliases lose entire result columns without a failed status. The declared manifest and delivered payload disagree.
Severity is assessed within MiniLake's documented local development/CI use; it does not assume a shared production Databricks deployment.
Suggested Fix
Build Arrow tables from ordered arrays plus a schema/names list, preserving each result position. Do not pass through a dictionary keyed by name.
Test Coverage Gaps
Format tests use unique aliases. Add duplicate names, repeated expressions, and joins with overlapping column names; compare column count and positional values across JSON/CSV/Arrow.
Severity
High
Confidence
High — reproduced through the real HTTP API and traced to the source paths above.
Recommended Labels
bug, high-priority
Existing Issue Matches
No reasonable duplicate found. Reviewed all open and closed issues and open pull requests in dmux/minilake, then searched component/function/symptom terms for this finding. The only open PR at review time, #29, changes the virtualenv lock entry and does not address this defect. The documented intentional limitations were checked before reporting.
Activity
eddiethedean commentedon Oct 25, 2026
Verified the duplicate-column fix in a frozen local source snapshot at
e24f46a7753ff0be30df3d83978c370c817f89cdthrough the real HTTP external-link download path, using MiniLake 1.9.0, DuckDB 1.5.6 and PyArrow 26.0.0.SELECT 1 AS same, 2 AS samewithformat=ARROW_STREAManddisposition=EXTERNAL_LINKSnow decodes to two columns namedsame, with positional column values[1]and[2]. This matches the ordered-array fix being tracked in #54; no duplicate issue is needed.The reproduction used a fresh statement cache so the independent cache-overflow defect in #33 could not mask the serializer result.