FrameQuant: Flexible Low-Bit Quantization for Transformers

arXiv:2403.06082v1 Announce Type: new
Abstract: Transformers are the backbone of powerful foundation models for many Vision and Natural Language Processing tasks. But their compute and memory/storage footprint is large, and so, serving such models is expensive often requiring high-end hardware. To mitigate this difficulty, Post-Training Quantization seeks to modify a pre-trained model and quantize it to eight bits or lower, significantly boosting compute/memory/latency efficiency. Such models have been successfully quantized to four bits with some performance loss. In this work, …

abstract and natural language processing arxiv compute cs.cl cs.lg foundation hardware language language processing low memory natural natural language natural language processing processing quantization storage tasks training transformers type vision