CHAI: Clustered Head Attention for Efficient LLM Inference

arXiv:2403.08058v1 Announce Type: new
Abstract: Large Language Models (LLMs) with hundreds of billions of parameters have transformed the field of machine learning. However, serving these models at inference time is both compute and memory intensive, where a single request can require multiple GPUs and tens of Gigabytes of memory. Multi-Head Attention is one of the key components of LLMs, which can account for over 50% of LLMs memory and compute requirement. We observe that there is a high amount of …

abstract arxiv attention chai compute cs.cl cs.lg gpus head however inference language language models large language large language models llm llms machine machine learning memory multi-head multi-head attention multiple parameters request type