Spark, Dask, DuckDB, Polars: TPC-H Benchmarks at Scale

I gave this talk at PyData NYC last week. It was fun working with devs from various projects (Dask, Arrow, Polars, Spark) in the week leading up to the event. Thought I'd share a re-recording of it here

[https://youtu.be/wKH0-zs2g\_U](https://youtu.be/wKH0-zs2g_U)

This is the result of a couple weeks of work comparing large data frameworks on benchmarks ranging in size 10GB to 10TB. No project wins. It's really interesting analyzing results though.

DuckDB and Dask are the only projects that reliably finish …

arrow benchmarks dask data datascience devs duckdb event frameworks fun nyc projects recording scale spark talk thought work