Limitations of neural network training due to numerical instability of backpropagation. (arXiv:2210.00805v2 [cs.LG] UPDATED)

We study the training of deep neural networks by gradient descent where
floating-point arithmetic is used to compute the gradients. In this framework
and under realistic assumptions, we demonstrate that it is highly unlikely to
find ReLU neural networks that maintain, in the course of training with
gradient descent, superlinearly many affine pieces with respect to their number
of layers. In virtually all approximation theoretical arguments which yield
high order polynomial rates of approximation, sequences of ReLU neural networks
with …

arxiv backpropagation network network training neural network numerical training