Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networks. (arXiv:2206.01278v1 [cs.LG])

A striking observation about iterative magnitude pruning (IMP; Frankle et al.
2020) is that $\unicode{x2014}$ after just a few hundred steps of dense
training $\unicode{x2014}$ the method can find a sparse sub-network that can be
trained to the same accuracy as the dense network. However, the same does not
hold at step 0, i.e. random initialization. In this work, we seek to understand
how this early phase of pre-training leads to a good initialization for IMP
both through the lens …

arxiv data diet networks