ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching

arXiv:2403.17312v1 Announce Type: cross
Abstract: The Transformer architecture has significantly advanced natural language processing (NLP) and has been foundational in developing large language models (LLMs) such as LLaMA and OPT, which have come to dominate a broad range of NLP tasks. Despite their superior accuracy, LLMs present unique challenges in practical inference, concerning the compute and memory-intensive nature. Thanks to the autoregressive characteristic of LLM inference, KV caching for the attention layers in Transformers can effectively accelerate LLM inference by …

abstract accuracy advanced architecture arxiv caching challenges cs.ai cs.lg cs.pf foundational inference language language model language models language processing large language large language model large language models llama llms natural natural language natural language processing nlp processing sparsity tasks transformer transformer architecture type via