SparQ Attention: Bandwidth-Efficient LLM Inference

arXiv:2312.04985v2 Announce Type: replace
Abstract: Generative large language models (LLMs) have opened up numerous novel possibilities, but due to their significant computational requirements their ubiquitous use remains challenging. Some of the most useful applications require processing large numbers of samples at a time and using long contexts, both significantly increasing the memory communication load of the models. We introduce SparQ Attention, a technique for increasing the inference throughput of LLMs by reducing the memory bandwidth requirements within the attention blocks …

abstract applications arxiv attention bandwidth communication computational cs.lg generative inference language language models large language large language models llm llms memory novel numbers processing requirements samples type