Training on large datasets

I have a really large dataset of data, which does not fit on the local disk. So I’m using S3 storing that.
But I see that I have a bottle neck on downloading part in the training pipeline. Looks like it limited by machine. So I want to use multiple machines in the same local network, which being a “preprocessing worker”, their task is download the data, preprocess them and push it into the task queue. Using of queue may …

datasets deeplearning large datasets training