ATP: Enabling Fast LLM Serving via Attention on Top Principal Keys

arXiv:2403.02352v1 Announce Type: new
Abstract: We propose a new attention mechanism with linear complexity, ATP, that fixates \textbf{A}ttention on \textbf{T}op \textbf{P}rincipal keys, rather than on each individual token. Particularly, ATP is driven by an important observation that input sequences are typically low-rank, i.e., input sequences can be represented by a few principal bases. Therefore, instead of directly iterating over all the input tokens, ATP transforms inputs into an orthogonal space and computes attention only on the top principal bases (keys). …

abstract arxiv attention complexity cs.ai cs.lg enabling keys linear llm low observation token type via