A Simple Finite-Time Analysis of TD Learning with Linear Function Approximation

arXiv:2403.02476v1 Announce Type: new
Abstract: We study the finite-time convergence of TD learning with linear function approximation under Markovian sampling. Existing proofs for this setting either assume a projection step in the algorithm to simplify the analysis, or require a fairly intricate argument to ensure stability of the iterates. We ask: \textit{Is it possible to retain the simplicity of a projection-based analysis without actually performing a projection step in the algorithm?} Our main contribution is to show this is possible …

abstract algorithm analysis approximation arxiv convergence cs.lg cs.sy eess.sy function linear math.oc projection sampling simple stability study the algorithm type