A Novel Hessian-Free Bilevel Optimizer via Evolution Strategies. (arXiv:2110.07004v2 [cs.LG] UPDATED)

Bilevel optimization has arisen as a powerful tool for solving many modern
machine learning problems. However, due to the nested structure of bilevel
optimization, even gradient-based methods require second-order derivative
approximations via Jacobian- or/and Hessian-vector computations, which can be
very costly in practice. In this work, we propose a novel Hessian-free bilevel
algorithm, which adopts the Evolution Strategies (ES) method to approximate the
response Jacobian matrix in the hypergradient of the bilevel problem, and hence
fully eliminates all second-order computations. …

arxiv evolution strategies