Enhanced Sparsification via Stimulative Training

arXiv:2403.06417v1 Announce Type: new
Abstract: Sparsification-based pruning has been an important category in model compression. Existing methods commonly set sparsity-inducing penalty terms to suppress the importance of dropped weights, which is regarded as the suppressed sparsification paradigm. However, this paradigm inactivates the dropped parts of networks causing capacity damage before pruning, thereby leading to performance degradation. To alleviate this issue, we first study and reveal the relative sparsity effect in emerging stimulative training and then propose a structured pruning framework, …

abstract arxiv capacity compression cs.cv however importance networks paradigm performance pruning set sparsity terms training type via