Castling-ViT: Compressing Self-Attention via Switching Towards Linear-Angular Attention During Vision Transformer Inference. (arXiv:2211.10526v1 [cs.CV])

Vision Transformers (ViTs) have shown impressive performance but still
require a high computation cost as compared to convolutional neural networks
(CNNs), due to the global similarity measurements and thus a quadratic
complexity with the input tokens. Existing efficient ViTs adopt local attention
(e.g., Swin) or linear attention (e.g., Performer), which sacrifice ViTs'
capabilities of capturing either global or local context. In this work, we ask
an important research question: Can ViTs learn both global and local context
while being more …

angular arxiv attention inference linear self-attention transformer vision