Convergence of Gradient Descent for Recurrent Neural Networks: A Nonasymptotic Analysis

arXiv:2402.12241v1 Announce Type: new
Abstract: We analyze recurrent neural networks trained with gradient descent in the supervised learning setting for dynamical systems, and prove that gradient descent can achieve optimality \emph{without} massive overparameterization. Our in-depth nonasymptotic analysis (i) provides sharp bounds on the network size $m$ and iteration complexity $\tau$ in terms of the sequence length $T$, sample size $n$ and ambient dimension $d$, and (ii) identifies the significant impact of long-term dependencies in the dynamical system on the convergence …

abstract analysis analyze arxiv complexity convergence cs.lg gradient iteration massive math.oc network networks neural networks prove recurrent neural networks stat.ml supervised learning systems type