Improved Powered Stochastic Optimization Algorithms for Large-Scale Machine Learning

Stochastic optimization, especially stochastic gradient descent (SGD), is now the workhorse for the vast majority of problems in machine learning. Various strategies, e.g., control variates, adaptive learning rate, momentum technique, etc., have been developed to improve canonical SGD that is of a low convergence rate and the poor generalization in practice. Most of these strategies improve SGD that can be attributed to control the updating direction (e.g., gradient descent or gradient ascent direction), or manipulate the learning rate. Along these …

algorithms canonical control convergence etc gradient low machine machine learning optimization practice rate scale stochastic strategies