Why Should Adam Optimizer Not Be the Default Learning Algorithm?

An increasing share of deep learning practitioners is training their models with adaptive gradient methods due to their rapid training time. Adam, in particular, has become the default algorithm used across many deep learning frameworks. Despite superior training outcomes, Adam and other adaptive optimization methods are known to generalize poorly compared to Stochastic gradient descent (SGD). These methods tend to perform well on the training data but are outperformed by SGD on the test data.

Lately, many researchers have penned …

adam adaptive-learning algorithm gradient-descent learning optimization-algorithms