Atom: Low-bit Quantization for Efficient and Accurate LLM Serving

arXiv:2310.19102v3 Announce Type: replace
Abstract: The growing demand for Large Language Models (LLMs) in applications such as content generation, intelligent chatbots, and sentiment analysis poses considerable challenges for LLM service providers. To efficiently use GPU resources and boost throughput, batching multiple requests has emerged as a popular paradigm; to further speed up batching, LLM quantization techniques reduce memory consumption and increase computing capacity. However, prevalent quantization schemes (e.g., 8-bit weight-activation quantization) cannot fully leverage the capabilities of modern GPUs, such …

abstract analysis applications arxiv atom batching boost challenges chatbots content generation cs.lg demand gpu gpu resources intelligent language language models large language large language models llm llms low multiple paradigm popular quantization resources sentiment sentiment analysis service service providers speed type