Repository navigation
[BUG] Arrow result serialization loses SQL types and binary bytes #42
Description
Description
Arrow output is built from values already converted for JSON, so dates become strings and binary values lose non-UTF-8 bytes. Empty/all-null columns are inferred as Arrow null instead of their SQL type.
Reproduced on MiniLake 1.9.0, commit fe8837dcfac9df621dfabc440abc07ebb4cdbcbb, using the repository's Docker test image, a real HTTP server, and fresh disposable state. Image Python sources were verified byte-for-byte against this commit. No mocks or production data were used.
Affected Components
- src/minilake/services/sql_statements.py —
_format_value - src/minilake/services/sql_statements.py —
_execute_sql_real_typed - src/minilake/services/sql_statements.py —
_serialize_chunk
Steps to Reproduce
Use a disposable instance and data directory. Paths below are relative to its HTTP base URL; bodies are JSON unless explicitly described as raw bytes. IDs shown as W/J/R/Q/C/HANDLE are replaced with the IDs returned by prior calls.
- Create a warehouse and execute
SELECT DATE '2024-01-01' AS d, from_hex('FF00') AS b, NULL::INTEGER AS nwithdisposition:"EXTERNAL_LINKS"andformat:"ARROW_STREAM". - Download the returned external link and inspect
pyarrow.ipc.open_stream(payload).read_all().schema. - Compare it with
manifest.schema.columns[0].type_text. - Follow-up regression inputs should include
SELECT NULL::INTEGER AS n, a typed empty result, DECIMAL/TIMESTAMP and BLOB values.
Expected Behavior
Arrow carries the native typed SQL result; DATE is an Arrow date, and declared types are retained even for null/empty results.
Actual Behavior
The manifest reports DATE, BLOB and INTEGER. The actual Arrow schema is d: string, b: string, n: null. DATE becomes a string, binary FF00 becomes a replacement character followed by NUL, and the typed null integer loses its declared type.
Root Cause Analysis
_execute_sql_real_typed always runs _format_value before format selection. This stringifies non-primitive values (and decodes bytes using replacement). _serialize_chunk then infers Arrow types from those altered Python values, ignoring raw_column_types.
Impact
Arrow clients receive schema/value types that disagree with SQL and the manifest, breaking typed downstream processing. Binary conversion is irreversible in this code path; the executed reproduction confirms DATE, BLOB and typed-null INTEGER defects.
Severity is assessed within MiniLake's documented local development/CI use; it does not assume a shared production Databricks deployment.
Suggested Fix
Keep a native/Arrow result representation until serialization. Build Arrow output using DuckDB Arrow export or an explicit schema with intact values; perform JSON formatting only for JSON delivery.
Test Coverage Gaps
Arrow tests only cover simple scalar values. Add DATE/TIMESTAMP/DECIMAL/BINARY, nested values, all-null and zero-row results, and assert actual Arrow types against the manifest. This issue concerns type preservation, separately from duplicate-column loss.
Severity
Medium
Confidence
High — reproduced through the real HTTP API and traced to the source paths above.
Recommended Labels
bug
Existing Issue Matches
No reasonable duplicate found. Reviewed all open and closed issues and open pull requests in dmux/minilake, then searched component/function/symptom terms for this finding. The only open PR at review time, #29, changes the virtualenv lock entry and does not address this defect. The documented intentional limitations were checked before reporting.
Activity