Switch EMA: A Free Lunch for Better Flatness and Sharpness

arXiv:2402.09240v1 Announce Type: new
Abstract: Exponential Moving Average (EMA) is a widely used weight averaging (WA) regularization to learn flat optima for better generalizations without extra cost in deep neural network (DNN) optimization. Despite achieving better flatness, existing WA methods might fall into worse final performances or require extra test-time computations. This work unveils the full potential of EMA with a single line of modification, i.e., switching the EMA parameters to the original model after each epoch, dubbed as Switch …

abstract arxiv cost cs.cv cs.lg deep neural network dnn extra free learn moving network neural network optimization performances regularization test type