On the Pros and Cons of Momentum Encoder in Self-Supervised Visual Representation Learning. (arXiv:2208.05744v1 [cs.CV])

Exponential Moving Average (EMA or momentum) is widely used in modern
self-supervised learning (SSL) approaches, such as MoCo, for enhancing
performance. We demonstrate that such momentum can also be plugged into
momentum-free SSL frameworks, such as SimCLR, for a performance boost. Despite
its wide use as a fundamental component in modern SSL frameworks, the benefit
caused by momentum is not well understood. We find that its success can be at
least partly attributed to the stability effect. In the first …

arxiv cons cv encoder learning pros representation representation learning