Transition Rate Scheduling for Quantization-Aware Training

arXiv:2404.19248v1 Announce Type: new
Abstract: Quantization-aware training (QAT) simulates a quantization process during training to lower bit-precision of weights/activations. It learns quantized weights indirectly by updating latent weights, i.e., full-precision inputs to a quantizer, using gradient-based optimizers. We claim that coupling a user-defined learning rate (LR) with these optimizers is sub-optimal for QAT. Quantized weights transit discrete levels of a quantizer, only if corresponding latent weights pass transition points, where the quantizer changes discrete states. This suggests that the changes …

abstract arxiv claim cs.cv gradient inputs precision process quantization rate scheduling training transition type