Benchmarks | Vortex

These summaries are complete without JavaScript. The charts need it.

Compression23 chartsencode 1.9× · decode 5.1× faster than Parquet

Write Throughput

#1vortex1.93x1981 MB/s

#2parquet1.00x1193 MB/s

#3lance0.67x750 MB/s

Scan Throughput

#1vortex5.08x22.65 GB/s

#2parquet1.00x4.42 GB/s

#3lance0.99x5.05 GB/s

Geomean throughput ratio to Parquet | Aggregate throughput (higher is better)

Compression Size32 charts23% larger than Parquet

vs Arrow

#1parquet7.1x

#2vortex5.8x

#3lance3.36x

#4arrow-ipc1.09x

vs ParquetMinMeanMax

Geometric means of compressed sizes versus Arrow (higher is better) and versus Parquet-zstd (lower is better)

String Encoding21 charts

Write Throughput

#1fsst1.00x223 MB/s

#2onpair-120.64x154 MB/s

#3onpair-160.62x149 MB/s

Scan Throughput

#1onpair-161.23x4.59 GB/s

#2onpair-121.20x4.60 GB/s

#3fsst1.00x3.72 GB/s

Geomean throughput ratio to FSST | Aggregate throughput (higher is better)

String Encoding Size21 charts

vs Arrow

#1onpair-162.76x

#2onpair-122.76x

#3fsst2.14x

vs FSSTMinMeanMax

Geometric means of file sizes versus Arrow (higher is better) and versus FSST (lower is better)

Random Access (NVMe)9 chartshot 81× · cold 26× faster than Parquet

Hot

#1vortex1.02x1.06 ms

#2litdata1.97x1.72 ms

#3lance2.50x3.05 ms

#4arrow-ipc4.44x16.38 ms

#5parquet82.44x78.05 ms

Cold

#1vortex1.44x3.41 ms

#2lance1.88x4.47 ms

#3arrow-ipc2.17x16.09 ms

#4litdata21.21x46.91 ms

#5parquet38.00x79.12 ms

Geomean of take time ratio to fastest across every dataset (lower is better)

Random Access (S3)9 chartshot 68× · cold 2.7× faster than Parquet

Hot

#1vortex1.00x22.46 ms

#2lance22.32x163.30 ms

#3parquet67.78x470.90 ms

Cold

#1vortex1.00x251.81 ms

#2lance2.34x585.03 ms

#3parquet2.67x670.64 ms

Geomean of take time ratio to fastest across every dataset (lower is better)

Clickbench43 chartsDuckDB 1.8× · DataFusion 1.6× faster than Parquet

Performance Summary

#1duckdb:vortex1.11x3.60 s

#2datafusion:vortex1.22x3.79 s

#3duckdb:vortex-compact1.32x4.20 s

#4datafusion:vortex-compact1.50x4.47 s

#5datafusion:parquet1.99x5.92 s

#6duckdb:duckdb2.02x5.40 s

#7duckdb:parquet2.04x5.63 s

#8datafusion:lance17.38x48.36 s

Geomean of query time ratio to fastest (lower is better)

TPC-H (NVMe) (SF=1)22 chartsDuckDB 3.6× · DataFusion 3.9× faster than Parquet

Performance Summary

#1duckdb:duckdb1.11x335.43 ms

#2duckdb:vortex1.29x383.37 ms

#3duckdb:vortex-compact1.42x415.42 ms

#4datafusion:vortex1.69x524.01 ms

#5datafusion:vortex-compact1.80x551.06 ms

#6datafusion:arrow3.29x1.08 s

#7duckdb:parquet4.70x1.39 s

#8datafusion:lance6.17x2.00 s

#9datafusion:parquet6.52x1.86 s

Geomean of query time ratio to fastest (lower is better)

TPC-H (S3) (SF=1)22 chartsDuckDB ≈ same · DataFusion 1.1× slower vs Parquet

Performance Summary

#1datafusion:parquet1.14x5.34 s

#2datafusion:vortex-compact1.21x5.66 s

#3datafusion:vortex1.31x6.13 s

#4duckdb:vortex-compact1.55x7.12 s

#5datafusion:lance1.57x7.69 s

#6duckdb:vortex1.63x7.50 s

#7duckdb:parquet1.64x7.67 s

Geomean of query time ratio to fastest (lower is better)

TPC-H (NVMe) (SF=10)22 chartsDuckDB 2.5× · DataFusion 2.3× faster than Parquet

Performance Summary

#1duckdb:duckdb1.14x1.02 s

#2duckdb:vortex1.30x1.16 s

#3duckdb:vortex-compact1.48x1.29 s

#4datafusion:vortex1.76x1.79 s

#5datafusion:vortex-compact1.95x1.90 s

#6duckdb:parquet3.23x2.89 s

#7datafusion:parquet4.13x3.60 s

#8datafusion:arrow14.54x16.39 s

#9datafusion:lance20.88x22.24 s

Geomean of query time ratio to fastest (lower is better)

TPC-H (S3) (SF=10)22 chartsDuckDB 1.6× slower · DataFusion ≈ same vs Parquet

Performance Summary

#1datafusion:vortex-compact1.10x11.13 s

#2datafusion:parquet1.18x12.40 s

#3datafusion:vortex1.23x12.64 s

#4duckdb:parquet1.35x13.60 s

#5duckdb:vortex-compact2.03x21.40 s

#6duckdb:vortex2.10x21.77 s

#7datafusion:lance3.53x43.02 s

Geomean of query time ratio to fastest (lower is better)

TPC-H (NVMe) (SF=100)22 chartsDuckDB 1.7× · DataFusion 2.0× faster than Parquet

Performance Summary

#1duckdb:vortex1.14x7.17 s

#2datafusion:vortex1.56x12.56 s

#3duckdb:parquet1.90x11.19 s

#4datafusion:parquet3.12x19.81 s

Geomean of query time ratio to fastest (lower is better)

TPC-H (S3) (SF=100)22 chartsDuckDB 4.2× slower · DataFusion 1.1× faster vs Parquet

Performance Summary

#1duckdb:parquet1.04x36.94 s

#2datafusion:vortex1.32x48.33 s

#3datafusion:parquet1.46x53.48 s

#4duckdb:vortex4.40x178.20 s

Geomean of query time ratio to fastest (lower is better)

TPC-DS (NVMe) (SF=1)99 chartsDuckDB 1.3× · DataFusion 1.5× faster than Parquet

Performance Summary

#1duckdb:duckdb1.02x1.69 s

#2duckdb:vortex1.36x2.11 s

#3duckdb:vortex-compact1.57x2.41 s

#4duckdb:parquet1.81x2.84 s

#5datafusion:vortex2.20x4.12 s

#6datafusion:vortex-compact2.51x4.86 s

#7datafusion:parquet3.27x6.04 s

Geomean of query time ratio to fastest (lower is better)

TPC-DS (NVMe) (SF=10)99 chartsDuckDB 1.5× · DataFusion 1.6× faster than Parquet

Performance Summary

#1duckdb:vortex1.05x4.99 s

#2datafusion:vortex1.41x9.66 s

#3duckdb:parquet1.54x6.98 s

#4datafusion:parquet2.30x13.88 s

Geomean of query time ratio to fastest (lower is better)

TPC-DS (NVMe) (SF=100)99 chartsDuckDB 1.2× · DataFusion 1.9× faster than Parquet

Performance Summary

#1duckdb:vortex1.16x22.61 s

#2duckdb:parquet1.43x27.02 s

#3datafusion:vortex1.45x121.08 s

#4datafusion:parquet2.83x147.91 s

Geomean of query time ratio to fastest (lower is better)

TPC-DS (S3) (SF=10)99 chartsDuckDB 1.2× slower · DataFusion ≈ same vs Parquet

Performance Summary

#1datafusion:vortex1.30x62.77 s

#2datafusion:parquet1.34x68.20 s

#3duckdb:parquet1.37x59.02 s

#4duckdb:vortex1.59x73.27 s

Geomean of query time ratio to fastest (lower is better)

TPC-DS (S3) (SF=100)99 chartsDuckDB 1.7× slower · DataFusion 1.9× faster vs Parquet

Performance Summary

#1datafusion:vortex1.18x234.43 s

#2duckdb:parquet1.37x165.96 s

#3datafusion:parquet2.19x367.54 s

#4duckdb:vortex2.32x335.57 s

Geomean of query time ratio to fastest (lower is better)

Statistical and Population Genetics11 chartsDuckDB 4.0× · DataFusion 3.9× faster than Parquet

Performance Summary

#1datafusion:vortex1.13x1.31 s

#2datafusion:vortex-compact1.23x1.50 s

#3duckdb:vortex2.80x2.67 s

#4duckdb:vortex-compact2.92x2.61 s

#5datafusion:parquet4.39x3.80 s

#6duckdb:parquet11.08x6.70 s

Geomean of query time ratio to fastest (lower is better)

Polar Signals Profiling11 chartsVortex only

Performance Summary

#1datafusion:vortex1.00x416.30 ms

Geomean of query time ratio to fastest (lower is better)

fineweb [nvme]9 chartsDuckDB 5.3× · DataFusion 3.4× faster than Parquet

Performance Summary

#1duckdb:vortex1.03x209.00 ms

#2datafusion:vortex1.32x219.82 ms

#3duckdb:vortex-compact3.16x733.66 ms

#4datafusion:parquet4.52x621.85 ms

#5datafusion:vortex-compact5.42x1.02 s

#6duckdb:parquet5.46x794.79 ms

Geomean of query time ratio to fastest (lower is better)

fineweb [s3]9 chartsDuckDB 1.3× · DataFusion 2.5× faster than Parquet

Performance Summary

#1datafusion:vortex1.03x2.89 s

#2datafusion:vortex-compact1.18x3.49 s

#3duckdb:vortex-compact1.63x5.05 s

#4duckdb:vortex1.85x5.83 s

#5duckdb:parquet2.34x5.69 s

#6datafusion:parquet2.58x6.16 s

Geomean of query time ratio to fastest (lower is better)

appian [nvme]8 chartsDuckDB 1.1× slower · DataFusion 1.3× faster vs Parquet

Performance Summary

#1duckdb:duckdb1.14x1.62 s

#2duckdb:parquet1.23x1.67 s

#3duckdb:vortex1.30x1.82 s

#4duckdb:vortex-compact1.31x1.84 s

#5datafusion:vortex1.88x11.00 s

#6datafusion:vortex-compact1.90x11.03 s

#7datafusion:parquet2.37x11.43 s

Geomean of query time ratio to fastest (lower is better)

clickbench-sorted [nvme]10 chartsDuckDB 2.1× · DataFusion 3.0× faster than Parquet

Performance Summary

#1duckdb:vortex1.09x195.31 ms

#2datafusion:vortex1.16x248.97 ms

#3duckdb:vortex-compact1.38x245.43 ms

#4datafusion:vortex-compact1.78x500.02 ms

#5duckdb:duckdb2.04x339.26 ms

#6duckdb:parquet2.25x441.89 ms

#7datafusion:parquet3.52x1.26 s

#8datafusion:lance78.20x20.17 s

Geomean of query time ratio to fastest (lower is better)

job [nvme]113 chartsDuckDB 3.8× faster than Parquet

Performance Summary

#1duckdb:duckdb1.08x4.83 s

#2duckdb:vortex1.27x5.51 s

#3duckdb:parquet4.88x20.25 s

Geomean of query time ratio to fastest (lower is better)

tpch-nested/list sf=1 [nvme]22 chartsDuckDB 7.5× · DataFusion 7.8× faster than Parquet

Performance Summary

#1duckdb:vortex1.14x758.39 ms

#2duckdb:duckdb1.22x788.77 ms

#3datafusion:vortex1.47x1.01 s

#4duckdb:vortex-compact1.48x963.47 ms

#5datafusion:vortex-compact1.73x1.16 s

#6datafusion:lance5.34x4.68 s

#7duckdb:parquet8.56x5.57 s

#8datafusion:parquet11.45x7.60 s

Geomean of query time ratio to fastest (lower is better)

vortex [nvme]3 chartsDuckDB 1.7× faster · DataFusion 1.4× slower vs Parquet

Performance Summary

#1datafusion:parquet1.15x492.71 ms

#2duckdb:vortex1.24x473.86 ms

#3datafusion:vortex1.65x648.59 ms

#4duckdb:parquet2.08x788.25 ms

Geomean of query time ratio to fastest (lower is better)