Latent Attention for Linear Time Transformers

arXiv:2402.17512v1 Announce Type: cross
Abstract: The time complexity of the standard attention mechanism in a transformer scales quadratically with the length of the sequence. We introduce a method to reduce this to linear scaling with time, based on defining attention via latent vectors. The method is readily usable as a drop-in replacement for the standard attention mechanism. Our "Latte Transformer" model can be implemented for both bidirectional and unidirectional tasks, with the causal version allowing a recurrent implementation which is …

abstract arxiv attention complexity cs.cl linear reduce replacement scaling standard stat.ml transformer transformers type vectors via