Simple linear attention language models balance the recall-throughput tradeoff

arXiv:2402.18668v1 Announce Type: cross
Abstract: Recent work has shown that attention-based language models excel at recall, the ability to ground generations in tokens previously seen in context. However, the efficiency of attention-based models is bottle-necked during inference by the KV-cache's aggressive memory consumption. In this work, we explore whether we can improve language model efficiency (e.g. by reducing memory consumption) without compromising on recall. By applying experiments and theory to a broad set of architectures, we identify a key tradeoff …

arxiv attention balance cs.cl cs.lg language language models linear recall simple type