LancBiO: dynamic Lanczos-aided bilevel optimization via Krylov subspace

arXiv:2404.03331v1 Announce Type: cross
Abstract: Bilevel optimization, with broad applications in machine learning, has an intricate hierarchical structure. Gradient-based methods have emerged as a common approach to large-scale bilevel problems. However, the computation of the hyper-gradient, which involves a Hessian inverse vector product, confines the efficiency and is regarded as a bottleneck. To circumvent the inverse, we construct a sequence of low-dimensional approximate Krylov subspaces with the aid of the Lanczos process. As a result, the constructed subspace is able …

abstract applications arxiv computation cs.lg dynamic efficiency gradient hierarchical however machine machine learning math.oc optimization product scale stat.ml type vector via