Dealing with unbounded gradients in stochastic saddle-point optimization

arXiv:2402.13903v1 Announce Type: new
Abstract: We study the performance of stochastic first-order methods for finding saddle points of convex-concave functions. A notorious challenge faced by such methods is that the gradients can grow arbitrarily large during optimization, which may result in instability and divergence. In this paper, we propose a simple and effective regularization technique that stabilizes the iterates and yields meaningful performance guarantees even if the domain and the gradient noise scales linearly with the size of the iterates …

abstract arxiv challenge cs.lg divergence functions math.oc optimization paper performance simple stat.ml stochastic study type