What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbation

arXiv:2403.06408v1 Announce Type: new
Abstract: Quantization has emerged as a promising technique for improving the memory and computational efficiency of large language models (LLMs). Though the trade-off between performance and efficiency is well-known, there is still much to be learned about the relationship between quantization and LLM performance. To shed light on this relationship, we propose a new perspective on quantization, viewing it as perturbations added to the weights and activations of LLMs. We call this approach "the lens of …

abstract arxiv computational cs.ai cs.lg efficiency language language models large language large language models llms memory performance quantization relationship study trade trade-off type