Versatile Single-Loop Method for Gradient Estimator: First and Second Order Optimality, and its Application to Federated Learning. (arXiv:2209.00361v1 [cs.LG])

While variance reduction methods have shown great success in solving large
scale optimization problems, many of them suffer from accumulated errors and,
therefore, should periodically require the full gradient computation. In this
paper, we present a single-loop algorithm named SLEDGE (Single-Loop mEthoD for
Gradient Estimator) for finite-sum nonconvex optimization, which does not
require periodic refresh of the gradient estimator but achieves nearly optimal
gradient complexity. Unlike existing methods, SLEDGE has the advantage of
versatility; (i) second-order optimality, (ii) exponential convergence …

application arxiv federated learning gradient learning loop