Compression-aware Training of Neural Networks using Frank-Wolfe

arXiv:2205.11921v2 Announce Type: replace
Abstract: Many existing Neural Network pruning approaches rely on either retraining or inducing a strong bias in order to converge to a sparse solution throughout training. A third paradigm, 'compression-aware' training, aims to obtain state-of-the-art dense models that are robust to a wide range of compression ratios using a single dense training run while also avoiding retraining. We propose a framework centered around a versatile family of norm constraints and the Stochastic Frank-Wolfe (SFW) algorithm that …

abstract art arxiv bias compression converge cs.lg math.oc network networks neural network neural networks paradigm pruning retraining robust solution state training type