Hidden Synergy: $L_1$ Weight Normalization and 1-Path-Norm Regularization

arXiv:2404.19112v1 Announce Type: new
Abstract: We present PSiLON Net, an MLP architecture that uses $L_1$ weight normalization for each weight vector and shares the length parameter across the layer. The 1-path-norm provides a bound for the Lipschitz constant of a neural network and reflects on its generalizability, and we show how PSiLON Net's design drastically simplifies the 1-path-norm, while providing an inductive bias towards efficient learning and near-sparse parameters. We propose a pruning method to achieve exact sparsity in the …

abstract architecture arxiv cs.lg hidden layer mlp network neural network norm normalization path regularization shares show stat.ml synergy type vector