[D] Best practices for storing multi-TB image datasets for use w/ PyTorch

Hello Everyone,

I'm working with a moderately large deep learning dataset (\~4.4 Tb) of satellite image data. Currently, I have the data stored as NPZ files. Each NPZ file contain the response labels and a time series of imagery.

After digging around, it seems like storing the data in HDF5 might be a better alternative and improve random read speed.

Does anyone have a suggestion for resources on best practices for managing large datasets? The information coming up on Google …

best practices data dataset datasets deep learning file files hello image image data image datasets labels machinelearning practices pytorch satellite series time series