Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learnt. (arXiv:2206.07137v3 [cs.LG] UPDATED)

Training on web-scale data can take months. But most computation and time is
wasted on redundant and noisy points that are already learnt or not learnable.
To accelerate training, we introduce Reducible Holdout Loss Selection
(RHO-LOSS), a simple but principled technique which selects approximately those
points for training that most reduce the model's generalization loss. As a
result, RHO-LOSS mitigates the weaknesses of existing data selection methods:
techniques from the optimization literature typically select 'hard' (e.g. high
loss) points, but …

arxiv training