Attention is Naturally Sparse with Gaussian Distributed Input

arXiv:2404.02690v1 Announce Type: new
Abstract: The computational intensity of Large Language Models (LLMs) is a critical bottleneck, primarily due to the $O(n^2)$ complexity of the attention mechanism in transformer architectures. Addressing this, sparse attention emerges as a key innovation, aiming to reduce computational load while maintaining model performance. This study presents a rigorous theoretical analysis of the sparsity in attention scores within LLMs, particularly under the framework of Gaussian inputs. By establishing a set of foundational assumptions and employing a …

abstract architectures arxiv attention complexity computational cs.ai cs.cl cs.lg distributed innovation intensity key language language models large language large language models llms performance reduce study transformer type