QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs

arXiv:2404.00456v1 Announce Type: new
Abstract: We introduce QuaRot, a new Quantization scheme based on Rotations, which is able to quantize LLMs end-to-end, including all weights, activations, and KV cache in 4 bits. QuaRot rotates LLMs in a way that removes outliers from the hidden state without changing the output, making quantization easier. This computational invariance is applied to the hidden state (residual) of the LLM, as well as to the activations of the feed-forward components, aspects of the attention mechanism …

arxiv cs.lg free inference llms outlier type