APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models

arXiv:2402.14866v1 Announce Type: new
Abstract: Large Language Models (LLMs) have greatly advanced the natural language processing paradigm. However, the high computational load and huge model sizes pose a grand challenge for deployment on edge devices. To this end, we propose APTQ (Attention-aware Post-Training Mixed-Precision Quantization) for LLMs, which considers not only the second-order information of each layer's weights, but also, for the first time, the nonlinear effect of attention outputs on the entire model. We leverage the Hessian trace as …

abstract advanced arxiv attention challenge computational cs.ai cs.cl cs.lg deployment devices edge edge devices language language models language processing large language large language models llms mixed mixed-precision natural natural language natural language processing paradigm precision processing quantization training type