Moonwalk: Inverse-Forward Differentiation

arXiv:2402.14212v1 Announce Type: new
Abstract: Backpropagation, while effective for gradient computation, falls short in addressing memory consumption, limiting scalability. This work explores forward-mode gradient computation as an alternative in invertible networks, showing its potential to reduce the memory footprint without substantial drawbacks. We introduce a novel technique based on a vector-inverse-Jacobian product that accelerates the computation of forward gradients while retaining the advantages of memory reduction and preserving the fidelity of true gradients. Our method, Moonwalk, has a time complexity …

abstract arxiv backpropagation computation consumption cs.ai cs.lg differentiation gradient memory memory consumption networks novel product reduce scalability type vector work