Randomized Forward Mode of Automatic Differentiation For Optimization Algorithms

We present a randomized forward mode gradient (RFG) as an alternative to backpropagation. RFG is a random estimator for the gradient that is constructed based on the directional derivative along a random vector. The forward mode automatic differentiation (AD) provides an efficient computation of RFG. The probability distribution of the random vector determines the statistical properties of RFG. Through the second moment analysis, we found that the distribution with the smallest kurtosis yields the smallest expected relative squared error. By …

algorithms backpropagation computation cs.ai cs.lg differentiation distribution gradient math.oc optimization probability random vector