Masks, Signs, And Learning Rate Rewinding

arXiv:2402.19262v1 Announce Type: new
Abstract: Learning Rate Rewinding (LRR) has been established as a strong variant of Iterative Magnitude Pruning (IMP) to find lottery tickets in deep overparameterized neural networks. While both iterative pruning schemes couple structure and parameter learning, understanding how LRR excels in both aspects can bring us closer to the design of more flexible deep learning algorithms that can optimize diverse sets of sparse architectures. To this end, we conduct experiments that disentangle the effect of mask …

abstract arxiv cs.lg design iterative masks networks neural networks pruning rate tickets type understanding