Fibottention: Inceptive Visual Representation Learning with Diverse Attention Across Heads

arXiv:2406.19391v1 Announce Type: new
Abstract: Visual perception tasks are predominantly solved by Vision Transformer (ViT) architectures, which, despite their effectiveness, encounter a computational bottleneck due to the quadratic complexity of computing self-attention. This inefficiency is largely due to the self-attention heads capturing redundant token interactions, reflecting inherent redundancy within visual data. Many works have aimed to reduce the computational complexity of self-attention in ViTs, leading to the development of efficient and sparse transformer architectures. In this paper, viewing through the …

abstract architectures arxiv attention complexity computational computing cs.cv diverse interactions perception redundancy representation representation learning self-attention tasks token transformer type vision visual vit