Global Optimality without Mixing Time Oracles in Average-reward RL via Multi-level Actor-Critic

arXiv:2403.11925v1 Announce Type: new
Abstract: In the context of average-reward reinforcement learning, the requirement for oracle knowledge of the mixing time, a measure of the duration a Markov chain under a fixed policy needs to achieve its stationary distribution-poses a significant challenge for the global convergence of policy gradient methods. This requirement is particularly problematic due to the difficulty and expense of estimating mixing time in environments with large state spaces, leading to the necessity of impractically long trajectories for …

abstract actor actor-critic arxiv challenge context convergence cs.lg distribution global knowledge markov oracle policy reinforcement reinforcement learning type via