An Improved Finite-time Analysis of Temporal Difference Learning with Deep Neural Networks

arXiv:2405.04017v1 Announce Type: new
Abstract: Temporal difference (TD) learning algorithms with neural network function parameterization have well-established empirical success in many practical large-scale reinforcement learning tasks. However, theoretical understanding of these algorithms remains challenging due to the nonlinearity of the action-value approximation. In this paper, we develop an improved non-asymptotic analysis of the neural TD method with a general $L$-layer neural network. New proof techniques are developed and an improved new $\tilde{\mathcal{O}}(\epsilon^{-1})$ sample complexity is derived. To our best knowledge, …

abstract algorithms analysis approximation arxiv cs.lg difference function however math.oc network networks neural network neural networks paper practical reinforcement reinforcement learning scale success tasks temporal type understanding value