Efficient Deep Learning with Decorrelated Backpropagation

arXiv:2405.02385v1 Announce Type: new
Abstract: The backpropagation algorithm remains the dominant and most successful method for training deep neural networks (DNNs). At the same time, training DNNs at scale comes at a significant computational cost and therefore a high carbon footprint. Converging evidence suggests that input decorrelation may speed up deep learning. However, to date, this has not yet translated into substantial improvements in training efficiency in large-scale DNNs. This is mainly caused by the challenge of enforcing fast and …

abstract algorithm arxiv backpropagation carbon carbon footprint computational cost cs.lg deep learning evidence however networks neural networks scale speed training type