QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving

arXiv:2405.04532v1 Announce Type: cross
Abstract: Quantization can accelerate large language model (LLM) inference. Going beyond INT8 quantization, the research community is actively exploring even lower precision, such as INT4. Nonetheless, state-of-the-art INT4 quantization techniques only accelerate low-batch, edge LLM inference, failing to deliver performance gains in large-batch, cloud-based LLM serving. We uncover a critical issue: existing INT4 quantization methods suffer from significant runtime overhead (20-90%) when dequantizing either weights or partial sums on GPUs. To address this challenge, we introduce …

arxiv cs.ai cs.cl cs.lg cs.pf design llm quantization type