Zeroth-Order Hard-Thresholding: Gradient Error vs. Expansivity

arXiv:2210.05279v2 Announce Type: replace
Abstract: $\ell_0$ constrained optimization is prevalent in machine learning, particularly for high-dimensional problems, because it is a fundamental approach to achieve sparse learning. Hard-thresholding gradient descent is a dominant technique to solve this problem. However, first-order gradients of the objective function may be either unavailable or expensive to calculate in a lot of real-world problems, where zeroth-order (ZO) gradients could be a good surrogate. Unfortunately, whether ZO gradients can work with the hard-thresholding operator is still …

abstract arxiv cs.lg error function gradient however machine machine learning math.oc optimization solve thresholding type