Robust Data Pruning: Uncovering and Overcoming Implicit Bias

arXiv:2404.05579v1 Announce Type: new
Abstract: In the era of exceptionally data-hungry models, careful selection of the training data is essential to mitigate the extensive costs of deep learning. Data pruning offers a solution by removing redundant or uninformative samples from the dataset, which yields faster convergence and improved neural scaling laws. However, little is known about its impact on classification bias of the trained models. We conduct the first systematic study of this effect and reveal that existing data pruning …

abstract arxiv bias convergence costs cs.cv cs.lg data data pruning dataset deep learning faster however laws pruning robust samples scaling solution training training data type