RTX 3060 12GB runs out of memory (keypoint-rcnn-resnet50, batch-size 8, 1280x720px)

I'm training a keypoint-rcnn-resnet50 with pytorch on a RTX 3060 12GB. Even with batch-size 8 the gpu runs out of memory right from the beginning.
What confuses me is that the image size is only 1280x720px. Is the overhead just normally that high or could there be something wrong with my code?

batch-size deeplearning gpu image memory normally pytorch resnet50 rtx rtx 3060 something training