A Full Adagrad algorithm with O(Nd) operations

arXiv:2405.01908v1 Announce Type: cross
Abstract: A novel approach is given to overcome the computational challenges of the full-matrix Adaptive Gradient algorithm (Full AdaGrad) in stochastic optimization. By developing a recursive method that estimates the inverse of the square root of the covariance of the gradient, alongside a streaming variant for parameter updates, the study offers efficient and practical algorithms for large-scale applications. This innovative strategy significantly reduces the complexity and resource demands typically associated with full-matrix methods, enabling more effective …

abstract algorithm arxiv challenges computational covariance gradient math.st matrix novel operations optimization recursive square stat.ml stat.th stochastic streaming type updates