Stacking as Accelerated Gradient Descent

arXiv:2403.04978v1 Announce Type: new
Abstract: Stacking, a heuristic technique for training deep residual networks by progressively increasing the number of layers and initializing new layers by copying parameters from older layers, has proven quite successful in improving the efficiency of training deep neural networks. In this paper, we propose a theoretical explanation for the efficacy of stacking: viz., stacking implements a form of Nesterov's accelerated gradient descent. The theory also covers simpler models such as the additive ensembles constructed in …

abstract arxiv cs.lg efficiency gradient networks neural networks paper parameters residual stat.ml training type