Why Gradient Clipping Methods Accelerate Training

Accelerated methods now have a theoretical justification

Optimization analysis is an active area of research and interest in machine learning. Many of us that have taken optimization classes have learned that there are accelerated optimization methods such as ADAM and RMSProp that outperform standard Gradient Descent on many tasks. Although these adaptive methods are popularly used, a theoretical justification of why they performed well on non-convex problems was not available, until recently. Today, …

gradient machine learning optimization paper-review training