BERT on a Data Diet: Finding Important Examples by Gradient-Based Pruning. (arXiv:2211.05610v1 [cs.CL])

Current pre-trained language models rely on large datasets for achieving
state-of-the-art performance. However, past research has shown that not all
examples in a dataset are equally important during training. In fact, it is
sometimes possible to prune a considerable fraction of the training set while
maintaining the test performance. Established on standard vision benchmarks,
two gradient-based scoring metrics for finding important examples are GraNd and
its estimated version, EL2N. In this work, we employ these two metrics for the
first …

arxiv bert data diet examples gradient pruning