[D] What framework do you do your data processing (small and large datasets) in?

Hey all!

I'm curious to know what everyone does their data processing in? For industry purposes on smaller datasets I've used Pandas and sklearn, while for larger ones I've used Dask.

• I know that Tensorflow and Pytorch have their own dataloader frameworks but does it scale to large datasets? Say 100GB++?

• Do people do data transformations in Jax?

• if you had infinite time how would you (re)do your company tech stack?

My data is 85% timeseries, 10% images, and 5% …

data data processing datasets framework machinelearning processing small