New logarithmic step size for stochastic gradient descent

arXiv:2404.01257v1 Announce Type: new
Abstract: In this paper, we propose a novel warm restart technique using a new logarithmic step size for the stochastic gradient descent (SGD) approach. For smooth and non-convex functions, we establish an $O(\frac{1}{\sqrt{T}})$ convergence rate for the SGD. We conduct a comprehensive implementation to demonstrate the efficiency of the newly proposed step size on the ~FashionMinst,~ CIFAR10, and CIFAR100 datasets. Moreover, we compare our results with nine other existing approaches and demonstrate that the new logarithmic …

abstract arxiv convergence cs.lg efficiency functions gradient implementation math.oc novel paper rate stochastic type warm