Towards Next-Level Post-Training Quantization of Hyper-Scale Transformers

arXiv:2402.08958v1 Announce Type: new
Abstract: With the increasing complexity of generative AI models, post-training quantization (PTQ) has emerged as a promising solution for deploying hyper-scale models on edge devices such as mobile devices and TVs. Existing PTQ schemes, however, consume considerable time and resources, which could be a bottleneck in real situations where frequent model updates and multiple hyper-parameter tunings are required. As a cost-effective alternative, one-shot PTQ schemes have been proposed. Still, the performance is somewhat limited because they …

abstract ai models arxiv complexity cs.ai cs.lg devices edge edge devices generative generative ai models mobile mobile devices next quantization resources scale solution training transformers type