Regularization-based Pruning of Irrelevant Weights in Deep Neural Architectures. (arXiv:2204.04977v2 [cs.CL] UPDATED)

Deep neural networks exploiting millions of parameters are nowadays the norm
in deep learning applications. This is a potential issue because of the great
amount of computational resources needed for training, and of the possible loss
of generalization performance of overparametrized networks. We propose in this
paper a method for learning sparse neural topologies via a regularization
technique which identifies non relevant weights and selectively shrinks their
norm, while performing a classic update for relevant ones. This technique,
which is …

architectures arxiv neural architectures pruning regularization