Flowformer: Linearizing Transformers with Conservation Flows. (arXiv:2202.06258v2 [cs.LG] UPDATED)

Web: http://arxiv.org/abs/2202.06258

Transformers based on the attention mechanism have achieved impressive
success in various areas. However, the attention mechanism has a quadratic
complexity, significantly impeding Transformers from dealing with numerous
tokens and scaling up to bigger models. Previous methods mainly utilize the
similarity decomposition and the associativity of matrix multiplication to
devise linear-time attention mechanisms. They avoid degeneration of attention
to a trivial distribution by reintroducing inductive biases such as the
locality, thereby at the expense of model generality and expressiveness. In …

arxiv conservation lg transformers