Zeroth-Order Hard-Thresholding: Gradient Error vs. Expansivity. (arXiv:2210.05279v1 [cs.LG])

$\ell_0$ constrained optimization is prevalent in machine learning,
particularly for high-dimensional problems, because it is a fundamental
approach to achieve sparse learning. Hard-thresholding gradient descent is a
dominant technique to solve this problem. However, first-order gradients of the
objective function may be either unavailable or expensive to calculate in a lot
of real-world problems, where zeroth-order (ZO) gradients could be a good
surrogate. Unfortunately, whether ZO gradients can work with the
hard-thresholding operator is still an unsolved problem. To solve …

arxiv error gradient thresholding