Forward Learning with Top-Down Feedback: Empirical and Analytical Characterization

arXiv:2302.05440v2 Announce Type: replace
Abstract: "Forward-only" algorithms, which train neural networks while avoiding a backward pass, have recently gained attention as a way of solving the biologically unrealistic aspects of backpropagation. Here, we first address compelling challenges related to the "forward-only" rules, which include reducing the performance gap with backpropagation and providing an analytical understanding of their dynamics. To this end, we show that the forward-only algorithm with top-down feedback is well-approximated by an "adaptive-feedback-alignment" algorithm, and we analytically track …

abstract algorithms arxiv attention backpropagation challenges cs.lg feedback gap networks neural networks performance rules train type