slow training

I dont get it.I can fit up to 256 images on single batch size of 3x448x448, im using amp, gradient clipping,gradient accumulation,not a single .item(), .cpu(), moving with .to(device), torch.backends.cudnn.deterministic = False, torch.backends.cudnn.benchmark = True, using pillow simd for reading images.There 250 batches, and still one epoch take 2h30minutes.It takes 30 seconds pre batch.It should be about 5 or less.

HOW?

deeplearning training