Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformers

arXiv:2405.05219v1 Announce Type: new
Abstract: Large Language Models (LLMs) have profoundly changed the world. Their self-attention mechanism is the key to the success of transformers in LLMs. However, the quadratic computational cost $O(n^2)$ to the length $n$ input sequence is the notorious obstacle for further improvement and scalability in the longer context. In this work, we leverage the convolution-like structure of attention matrices to develop an efficient approximation method for attention computation using convolution matrices. We propose a $\mathsf{conv}$ basis …

abstract arxiv attention computation computational cost cs.ai cs.lg gradient however improvement inference key language language models large language large language models llms new paradigm paradigm self-attention success the key transformers type world