SimA: Simple Softmax-free Attention for Vision Transformers

arXiv:2206.08898v2 Announce Type: replace
Abstract: Recently, vision transformers have become very popular. However, deploying them in many applications is computationally expensive partly due to the Softmax layer in the attention block. We introduce a simple but effective, Softmax-free attention block, SimA, which normalizes query and key matrices with simple $\ell_1$-norm instead of using Softmax layer. Then, the attention block in SimA is a simple multiplication of three matrices, so SimA can dynamically change the ordering of the computation at the …

arxiv attention cs.cv free sima simple softmax transformers type vision vision transformers