Implicit Bias of AdamW: $\ell_\infty$ Norm Constrained Optimization

arXiv:2404.04454v1 Announce Type: new
Abstract: Adam with decoupled weight decay, also known as AdamW, is widely acclaimed for its superior performance in language modeling tasks, surpassing Adam with $\ell_2$ regularization in terms of generalization and optimization. However, this advantage is not theoretically well-understood. One challenge here is that though intuitively Adam with $\ell_2$ regularization optimizes the $\ell_2$ regularized loss, it is not clear if AdamW optimizes a specific objective. In this work, we make progress toward understanding the benefit of …

abstract adam arxiv bias challenge cs.lg however language math.oc modeling norm optimization performance regularization stat.ml tasks terms type