How Do Vision Transformers Work?. (arXiv:2202.06709v3 [cs.CV] UPDATED)

Web: http://arxiv.org/abs/2202.06709

The success of multi-head self-attentions (MSAs) for computer vision is now
indisputable. However, little is known about how MSAs work. We present
fundamental explanations to help better understand the nature of MSAs. In
particular, we demonstrate the following properties of MSAs and Vision
Transformers (ViTs): (1) MSAs improve not only accuracy but also generalization
by flattening the loss landscapes. Such improvement is primarily attributable
to their data specificity, not long-range dependency. On the other hand, ViTs
suffer from non-convex losses. …

arxiv cv transformers vision work