Continue training on specific samples or entire dataset

I’m training depth prediction on a large dataset, the average loss has come down to 0.03 but there’re 50k samples with loss ranging from 0.1 to 1. Should I train on these 50k samples specifically(maybe it’ll save some time?) or should I keep training on the entire dataset?

dataset deeplearning training