A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergence

Modern policy optimization methods in reinforcement learning, such as TRPO and PPO, owe their success to the use of parameterized policies. However, while theoretical guarantees have been established for this class of algorithms, especially in the tabular setting, the use of general parameterization schemes remains mostly unjustified. In this work, we introduce a novel framework for policy optimization based on mirror descent that naturally accommodates general parameterizations. The policy class induced by our scheme recovers known classes, e.g., softmax, and …

algorithms class convergence cs.lg framework general linear math.oc math.st modern novel optimization policy ppo reinforcement reinforcement learning stat.ml stat.th success tabular