Decoupled Weight Decay for Any $p$ Norm

arXiv:2404.10824v2 Announce Type: replace
Abstract: With the success of deep neural networks (NNs) in a variety of domains, the computational and storage requirements for training and deploying large NNs have become a bottleneck for further improvements. Sparsification has consequently emerged as a leading approach to tackle these issues. In this work, we consider a simple yet effective approach to sparsification, based on the Bridge, or $L_p$ regularization during training. We introduce a novel weight decay scheme, which generalizes the standard …

abstract arxiv become computational cs.ai cs.lg cs.ne domains improvements math.oc networks neural networks nns norm requirements storage success training type work