Apache DataFusion

Do you love Andy Grove? Well, based on this week's episode in This Week in Open Lakehouse you'll get to hear first hand me and Scott Haines gushing about his work in the latest Apache DataFusion Ballista and Comet releases; along with Unity Catalog's new capabilities and Apache Parquet vector and dec float types proposals ;) https://lnkd.in/dGdnBeyR

Do you love Andy Grove? Well, based on this week's episode in This Week in Open Lakehouse you'll get to hear first hand me and Scott Haines gushing about his work in the latest Apache DataFusion Ballista and Comet releases; along with Unity Catalog's new capabilities and Apache Parquet vector and dec float types proposals ;) https://lnkd.in/dGdnBeyR

Apache DataFusion Ballista 55.0.0 is out 🎉 The main story of this release is less about new features and more about fixing the foundations. For a long time, Ballista had architectural limits that made it hard to run reliably at scale. This release addresses many of them: - Adaptive Query Execution is now on by default, so the scheduler replans between stages using real row counts instead of pre-execution estimates. All 99 TPC-DS queries now complete and match single-process DataFusion. - Tasks can now cover multiple input partitions instead of one task per partition, which cuts planning, serialization, and dispatch overhead. - Executors use a bounded memory pool with spilling by default, rather than growing until the container gets killed. - The shuffle has been reworked, and several failure paths now retry or fail promptly instead of leaving jobs hanging. With that in place, we ran TPC-H at SF1000 (Parquet on S3) on the same Kubernetes cluster of 32 executors (8 vCPU, 64 GiB each): - Spark 4.1.3: 898 s - Ballista 55.0.0: 502 s - Spark 4.1.3 + Comet 1.1.0: 455 s So Ballista is about 1.8x faster than Spark across the suite and faster on 19 of 22 queries. Comet is still about 10% faster than Ballista overall, though Ballista wins on 7 queries. These are not tuned comparisons; the benchmarking guide has the full configuration and per-query results. People sometimes ask why Apache DataFusion has two distributed subprojects, Comet and Ballista. They are for different audiences. Comet accelerates existing Spark workloads and aims to produce the same results as Spark, so you keep your jobs, APIs, and ecosystem. Ballista does not aim for Spark compatibility. It is for people who want distributed DataFusion queries with no JVM dependency. There is plenty more in the release, including an Arrow Flight SQL frontend for ADBC clients, durable event logs with a history server, and range shuffles for distributed window functions. Now that the architecture is in better shape, I expect we will see significant performance gains in upcoming releases. Thanks to all 25 contributors! https://lnkd.in/gxdemuNS

People have built distributed engines with Apache DataFusion for many years, and the DataFusion core has always included the lowest level pieces like plan serialization and FlightSQL integration. However, it took a long time to determine what else was common (it wasn't Spark style stages, it wasn't a particular service discovery mechanism). Instead, it turns out the reusable parts are dividing up partitions across multiple executors and shuffle operators. The DataFusion Distributed https://lnkd.in/gkf5hDYC project beautifully captures these common components and has a very clean and easy to use API. Kudos to Gabriel Musat Mestre and the rest of the team for this great library

Apache DataFusion Comet 1.1.0 has been released. It consists of ~400 commits from 40 contributors over about seven weeks. The project has become much more active in recent months and it is great to see so many regular contributors. Highlights: - Native Iceberg writes via iceberg-rust (experimental, off by default) - Native memory that the pool doesn't track is now measured and logged, which helps with sizing memory overhead and avoiding container OOM kills - Several memory pool accounting fixes - Native Parquet writes on Spark 4.0+ now use Spark's own write path (experimental) - Broader S3 credential support, including STS throttling protection on EKS - Shuffle performance fixes, plus native dynamic filter pushdown - Many correctness fixes where Comet differed from Spark Thanks to everyone who contributed! https://lnkd.in/gYxTE-2b #ApacheSpark #DataFusion #ApacheArrow #Rust #Iceberg

I'm not a huge fan of the JVM personally. But I can't deny its reach. It's everywhere, especially in enterprise data stacks. So at Columnar we've taken extra care to make sure ADBC works great with JVM languages. This week we added Scala to our ADBC quickstarts, alongside Java and Kotlin. The quickstarts are runnable examples that show you how to connect to a database with ADBC, run a query, and get the results back in Apache Arrow format. There are quickstarts for 40+ databases, from DuckDB and Apache DataFusion to Snowflake, ClickHouse, Databricks, and many more. And if the JVM isn't your thing, ADBC works in 8+ non-JVM languages too. Link to the quickstarts in the comments.

Just under 24 hours until the next Apache DataFusion Community Showcase. Join in for talks and demos from Hotdata's Shefeek Jinnah and Veeva Systems' Bruce Ritchie. Details and registration: https://hubs.ly/Q04yL8vv0 Thanks to Spice AI Luke Kim Wyatt Wenzel Andrew Lamb and InfluxData for highlighting the latest and greatest built on DataFusion!

Join us for the Apache DataFusion Community showcase where Shefeek Jinnah will be presenting Multi-catalog support in DataFusion-DuckLake which is core to the Hotdata Inc architecture. cc Eddie A. Tejeda Zac Farrell Anoop Narang Rohan D'souza