One-Step Forward and Backtrack: Overcoming Zig-Zagging in Loss-Aware Quantization Training

Weight quantization is an effective technique to compress deep neural networks for their deployment on edge devices with limited resources. Traditional loss-aware quantization methods commonly use the quantized gradient to replace the full-precision gradient. However, we discover that the gradient error will lead to an unexpected zig-zagging-like issue in the gradient descent learning procedures, where the gradient directions rapidly oscillate or zig-zag, and such issue seriously slows down the model convergence. Accordingly, this paper proposes a one-step forward and backtrack …

cs.lg deployment devices edge edge devices error gradient issue loss networks neural networks precision quantization resources training will