Sparse Weight Averaging with Multiple Particles for Iterative Magnitude Pruning

arXiv:2305.14852v2 Announce Type: replace
Abstract: Given the ever-increasing size of modern neural networks, the significance of sparse architectures has surged due to their accelerated inference speeds and minimal memory demands. When it comes to global pruning techniques, Iterative Magnitude Pruning (IMP) still stands as a state-of-the-art algorithm despite its simple nature, particularly in extremely sparse regimes. In light of the recent finding that the two successive matching IMP solutions are linearly connected without a loss barrier, we propose Sparse Weight …

abstract algorithm architectures art arxiv cs.ai cs.lg ever global inference iterative memory modern multiple networks neural networks pruning significance simple state type