Smoothing the Edges: Smooth Optimization for Sparse Regularization using Hadamard Overparametrization

arXiv:2307.03571v3 Announce Type: replace
Abstract: We present a framework for smooth optimization of explicitly regularized objectives for (structured) sparsity. These non-smooth and possibly non-convex problems typically rely on solvers tailored to specific models and regularizers. In contrast, our method enables fully differentiable and approximation-free optimization and is thus compatible with the ubiquitous gradient descent paradigm in deep learning. The proposed optimization transfer comprises an overparameterization of selected parameters and a change of penalties. In the overparametrized problem, smooth surrogate regularization …

abstract approximation arxiv contrast cs.lg differentiable framework free math.oc optimization regularization sparsity stat.ml type