Central Limit Theorem for Two-Timescale Stochastic Approximation with Markovian Noise: Theory and Applications

arXiv:2401.09339v2 Announce Type: replace-cross
Abstract: Two-timescale stochastic approximation (TTSA) is among the most general frameworks for iterative stochastic algorithms. This includes well-known stochastic optimization methods such as SGD variants and those designed for bilevel or minimax problems, as well as reinforcement learning like the family of gradient-based temporal difference (GTD) algorithms. In this paper, we conduct an in-depth asymptotic analysis of TTSA under controlled Markovian noise via central limit theorem (CLT), uncovering the coupled dynamics of TTSA influenced by the …

abstract algorithms applications approximation arxiv cs.lg family frameworks general gradient iterative math.oc minimax noise optimization reinforcement reinforcement learning stat.ml stochastic theorem theory timescale type variants