Maxwell's Demon at Work: Efficient Pruning by Leveraging Saturation of Neurons

arXiv:2403.07688v1 Announce Type: new
Abstract: When training deep neural networks, the phenomenon of $\textit{dying neurons}$ $\unicode{x2013}$units that become inactive or saturated, output zero during training$\unicode{x2013}$ has traditionally been viewed as undesirable, linked with optimization challenges, and contributing to plasticity loss in continual learning scenarios. In this paper, we reassess this phenomenon, focusing on sparsity and pruning. By systematically exploring the impact of various hyperparameter configurations on dying neurons, we unveil their potential to facilitate simple yet effective structured pruning algorithms. …

abstract arxiv become challenges continual cs.ai cs.lg loss networks neural networks neurons optimization paper pruning training type unicode units work