Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models

arXiv:2309.15531v2 Announce Type: replace
Abstract: Large Language Models (LLMs) have recently demonstrated remarkable success across various tasks. However, efficiently serving LLMs has been a challenge due to the large memory bottleneck, specifically in small batch inference settings (e.g. mobile devices). Weight-only quantization can be a promising approach, but sub-4 bit quantization remains a challenge due to large-magnitude activation outliers. To mitigate the undesirable outlier effect, we first propose per-IC quantization, a simple yet effective method that creates quantization groups within …

arxiv cs.lg dimensions language language models large language large language models low outliers quantization type