Custom Gradient Estimators are Straight-Through Estimators in Disguise

arXiv:2405.05171v1 Announce Type: new
Abstract: Quantization-aware training comes with a fundamental challenge: the derivative of quantization functions such as rounding are zero almost everywhere and nonexistent elsewhere. Various differentiable approximations of quantization functions have been proposed to address this issue. In this paper, we prove that when the learning rate is sufficiently small, a large class of weight gradient estimators is equivalent with the straight through estimator (STE). Specifically, after swapping in the STE and adjusting both the weight initialization …

abstract arxiv challenge cs.lg differentiable functions fundamental gradient issue paper prove quantization rate through training type