GPU stuck at 100 during PyTorch training loop.

Naturally during the training loop, the GPU should increase as the model starts working. What usually happens is that the GPU is low while the dataloader gets a batch and loads it to memory from disc. Then when the data is loaded the model begins working and the GPU usage increases significantly. When this is done, the GPU usage drops again while getting the next batch.

Once in a while however the GPU sticks at 100%, the system slows down, …

data dataloader deeplearning gpu loop low memory pytorch training usage