SGD with Clipping is Secretly Estimating the Median Gradient

arXiv:2402.12828v1 Announce Type: cross
Abstract: There are several applications of stochastic optimization where one can benefit from a robust estimate of the gradient. For example, domains such as distributed learning with corrupted nodes, the presence of large outliers in the training data, learning under privacy constraints, or even heavy-tailed noise due to the dynamics of the algorithm itself. Here we study SGD with robust gradient estimators based on estimating the median. We first consider computing the median gradient across samples, …

abstract applications arxiv benefit constraints cs.lg data distributed distributed learning domains example gradient math.oc nodes noise optimization outliers privacy robust stat.ml stochastic training training data type