Solving Attention Kernel Regression Problem via Pre-conditioner

arXiv:2308.14304v2 Announce Type: replace
Abstract: The attention mechanism is the key to large language models, and the attention matrix serves as an algorithmic and computational bottleneck for such a scheme. In this paper, we define two problems, motivated by designing fast algorithms for proxy of attention matrix and solving regressions against them. Given an input matrix $A\in \mathbb{R}^{n\times d}$ with $n\gg d$ and a response vector $b$, we first consider the matrix exponential of the matrix $A^\top A$ as a …

abstract algorithms arxiv attention computational cs.lg designing kernel key language language models large language large language models matrix paper regression the key them type via