One-Step Forward and Backtrack: Overcoming Zig-Zagging in Loss-Aware Quantization Training. (arXiv:2401.16760v1 [cs.LG])

Weight quantization is an effective technique to compress deep neural
networks for their deployment on edge devices with limited resources.
Traditional loss-aware quantization methods commonly use the quantized gradient
to replace the full-precision gradient. However, we discover that the gradient
error will lead to an unexpected zig-zagging-like issue in the gradient descent
learning procedures, where the gradient directions rapidly oscillate or
zig-zag, and such issue seriously slows down the model convergence.
Accordingly, this paper proposes a one-step forward and backtrack …

arxiv cs.lg deployment devices edge edge devices error gradient issue loss networks neural networks precision quantization resources training will