How do you compare different deep learning experiments when determinism is so hard to achieve?

Hello everyone! I'm working on a deep learning model that uses Resnet50 as a backbone and it takes around 5 days to fully train on my dataset. I'm now trying to optimise a hyperparameter, which means training the model for 10 times, so that I can pick the best one.

The problem that I'm facing is that training is non deterministic. First, because tensorflow on GPU is non deterministic, and enabling determinism means that tensorflow runs *100 times* slower. Second, …

dataset deep learning deeplearning hello hyperparameter resnet50 train training