Finite-time analysis of single-timescale actor-critic. (arXiv:2210.09921v1 [cs.LG])

Despite the great empirical success of actor-critic methods, its finite-time
convergence is still poorly understood in its most practical form. In
particular, the analysis of single-timescale actor-critic presents significant
challenges due to the highly inaccurate critic estimation and the complex error
propagation dynamics over iterations. Existing works on analyzing
single-timescale actor-critic only focus on the i.i.d. sampling or tabular
setting for simplicity, which is rarely the case in practical applications. We
consider the more practical online single-timescale actor-critic algorithm on …

actor-critic analysis arxiv