On the Compressibility of Quantized Large Language Models

arXiv:2403.01384v1 Announce Type: new
Abstract: Deploying Large Language Models (LLMs) on edge or mobile devices offers significant benefits, such as enhanced data privacy and real-time processing capabilities. However, it also faces critical challenges due to the substantial memory requirement of LLMs. Quantization is an effective way of reducing the model size while maintaining good performance. However, even after quantization, LLMs may still be too big to fit entirely into the limited memory of edge or mobile devices and have to …

abstract arxiv benefits capabilities challenges cs.ai cs.cl cs.lg data data privacy devices edge language language models large language large language models llms memory mobile mobile devices privacy processing quantization real-time real-time processing type