Repository navigation
Proposal: unify DuckDB and Spark table storage with Delta Lake #89
Description
Decision requested
Would you be open to a feasibility milestone for making persistent managed tables Delta-backed, so DuckDB SQL and PySpark can use the same rows through minilake's existing Unity Catalog API?
This requests agreement on the direction and initial investigation. It does not request an immediate default change or migration of existing users.
Problem
Managed tables currently live inside DuckDB and are unavailable to Spark. External Delta tables already share files across both engines. A workflow that creates a managed table through SQL/SDK and then processes it with Spark therefore needs an external-table setup or a separate exported copy.
Proposed direction
- Keep DuckDB as the SQL engine; store persistent user tables in shared Delta files.
- Let MANAGED versus EXTERNAL describe file ownership and lifecycle.
- Require baseline SQL, including UPDATE and DELETE, to work through an embedded writer without Docker, Java, or PySpark.
- Introduce an optional mode before considering a mandatory switch, recommended for minilake 2.0.0.
Main tradeoffs and safeguards
Native DuckDB can be faster, especially for short queries and joins. The proposal sets discussion targets for reads, writes, startup, and a complete SDK workflow; these are not measured claims.
The work also requires durable catalog metadata, a complete workspace migration, explicit rollback limits, and safe deferred file cleanup. External locations must remain separate from directories minilake can delete. The proposal retains the current Spark/connector line and DuckDB below 2.0, subject to compatibility testing.
Proposed first milestone
- Select and test an embedded writer for creation, CTAS, INSERT, UPDATE, and DELETE.
- Verify that the pinned Spark Unity Catalog connector can read and write tables advertised as MANAGED.
- Publish reproducible native-versus-Delta benchmarks and a supported-feature/version matrix.
- Bring those results back for a decision before implementing the default switch.
Full proposal
- Proposal, tradeoffs, performance targets, and release plan
- Implementation notes: ownership, consistency, full-workspace migration, and validation
Documentation-only review: pull request #90. Both proposal documents link back to this issue and that pull request. There are no runtime or dependency changes in this submission.
Activity