Omega: Optimistic EMA Gradients

arXiv:2306.07905v2 Announce Type: replace
Abstract: Stochastic min-max optimization has gained interest in the machine learning community with the advancements in GANs and adversarial training. Although game optimization is fairly well understood in the deterministic setting, some issues persist in the stochastic regime. Recent work has shown that stochastic gradient descent-ascent methods such as the optimistic gradient are highly sensitive to noise or can fail to converge. Although alternative strategies exist, they can be prohibitively expensive. We introduce Omega, a method …

abstract adversarial adversarial training arxiv community cs.lg ema game gans gradient machine machine learning math.oc max optimization stat.ml stochastic training type work