Forward Gradient-Based Frank-Wolfe Optimization for Memory Efficient Deep Neural Network Training

arXiv:2403.12511v1 Announce Type: new
Abstract: Training a deep neural network using gradient-based methods necessitates the calculation of gradients at each level. However, using backpropagation or reverse mode differentiation, to calculate the gradients necessities significant memory consumption, rendering backpropagation an inefficient method for computing gradients. This paper focuses on analyzing the performance of the well-known Frank-Wolfe algorithm, a.k.a. conditional gradient algorithm by having access to the forward mode of automatic differentiation to compute gradients. We provide in-depth technical details that show …

abstract arxiv backpropagation computing consumption cs.lg deep neural network differentiation frank gradient however math.oc memory memory consumption network network training neural network optimization paper rendering training type