A Novel Sparse Regularizer. (arXiv:2301.07285v5 [cs.LG] UPDATED)

$L_p$-norm regularization schemes such as $L_0$, $L_1$, and $L_2$-norm
regularization and $L_p$-norm-based regularization techniques such as weight
decay, LASSO, and elastic net compute a quantity which depends on model weights
considered in isolation from one another. This paper introduces a regularizer
based on minimizing a novel measure of entropy applied to the model during
optimization. In contrast with $L_p$-norm-based regularization, this
regularizer is concerned with the spatial arrangement of weights within a
weight matrix. This novel regularizer is an additive …

arxiv compute entropy function lasso loss matrix novel optimization paper regularization weight matrix