The Implicit Bias of Gradient Noise: A Symmetry Perspective

We characterize the learning dynamics of stochastic gradient descent (SGD) when continuous symmetry exists in the loss function, where the divergence between SGD and gradient descent is dramatic. We show that depending on how the symmetry affects the learning dynamics, we can divide a family of symmetry into two classes. For one class of symmetry, SGD naturally converges to solutions that have a balanced and aligned gradient noise. For the other class of symmetry, SGD will almost always diverge. Then, …

bias continuous cs.lg divergence dynamics family function gradient loss math.oc noise perspective show stat.ml stochastic symmetry