Massive drop in GPU usage

Hi!

I am trying to improve the memory and speed efficiency of our Pytorch training pipeline. During the inspection I realized our GPU becomes IDLE after every epoch (Visual at the end).

Our environment is:

* 2X V100 (Azure Cloud).
* Pytorch 1.13.
* CUDA 11.6.
* AMP is activated.
* Number of workers is 8.
* DataParallel is used.
* Batch size is 32.
* Pin memory set.
* Drop last set.
* Persistent workers set.
* We are …

azure azure cloud cloud cuda deeplearning efficiency environment gpu massive memory pipeline pytorch speed training usage v100 workers