Why Pandas-like Interfaces are Sub-optimal for Distributed Computing

A deep look at the assumptions of the Pandas interface

Written by Kevin Kho and Han Wang

This is a written version of our most recent PyCon talk.

Pandas-like Frameworks for Distributed Computing

Over the last year and a half, we’ve talked to data practitioners who want to move Pandas code to either Dask or Spark to take advantage of distributed computing resources. Their workloads were quickly becoming too compute-intense or their datasets …

computing dask data science distributed distributed computing distributed systems interfaces pandas spark