Gauss-Newton Temporal Difference Learning with Nonlinear Function Approximation

arXiv:2302.13087v2 Announce Type: replace-cross
Abstract: In this paper, a Gauss-Newton Temporal Difference (GNTD) learning method is proposed to solve the Q-learning problem with nonlinear function approximation. In each iteration, our method takes one Gauss-Newton (GN) step to optimize a variant of Mean-Squared Bellman Error (MSBE), where target networks are adopted to avoid double sampling. Inexact GN steps are analyzed so that one can safely and efficiently compute the GN updates by cheap matrix iterations. Under mild conditions, non-asymptotic finite-sample convergence …

abstract approximation arxiv cs.lg difference error function gauss iteration math.oc mean networks paper q-learning solve temporal type