Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging

arXiv:2306.16788v3 Announce Type: replace
Abstract: Neural networks can be significantly compressed by pruning, yielding sparse models with reduced storage and computational demands while preserving predictive performance. Model soups (Wortsman et al., 2022) enhance generalization and out-of-distribution (OOD) performance by averaging the parameters of multiple models into a single one, without increasing inference time. However, achieving both sparsity and parameter averaging is challenging as averaging arbitrary sparse models reduces the overall sparsity due to differing sparse connectivities. This work addresses these …

abstract arxiv computational cs.ai cs.lg distribution multiple networks neural networks parameters performance predictive pruning recipe storage type via