Attention-aware Post-training Quantization without Backpropagation

arXiv:2406.13474v1 Announce Type: new
Abstract: Quantization is a promising solution for deploying large-scale language models (LLMs) on resource-constrained devices. Existing quantization approaches, however, rely on gradient-based optimization, regardless of it being post-training quantization (PTQ) or quantization-aware training (QAT), which becomes problematic for hyper-scale LLMs with billions of parameters. This overhead can be alleviated via recently proposed backpropagation-free PTQ methods; however, their performance is somewhat limited by their lack of consideration of inter-layer dependencies. In this paper, we thus propose a …

abstract arxiv attention backpropagation cs.ai cs.lg deploying devices gradient however language language models llms optimization parameters quantization scale solution training type via