[R] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models - Massachusetts Institute of Technology and NVIDIA Guangxuan Xiao et al - Enables INT8 for LLM bigger than 100B parameters including OPT-175B, BLOOM-176B and

Paper: [https://arxiv.org/abs/2211.10438](https://arxiv.org/abs/2211.10438)

Github: [https://github.com/mit-han-lab/smoothquant](https://github.com/mit-han-lab/smoothquant)

Abstract:

Large language models (LLMs) show excellent performance but are compute- and memory-intensive. Quantization can reduce memory and accelerate inference. However, for LLMs beyond 100 billion parameters, existing methods cannot maintain accuracy or do not run efficiently on hardware. We propose SmoothQuant, a training-free, accuracy-preserving, and general-purpose post-training quantization (PTQ) solution to enable 8-bit weight, 8-bit activation (W8A8) quantization for LLMs that can be implemented efficiently. We observe that systematic outliers appear at fixed activation channels. …

bigger bloom language language models large language models llm machinelearning massachusetts nvidia opt-175b quantization technology training