PPO-Clip Attains Global Optimality: Towards Deeper Understandings of Clipping

arXiv:2312.12065v2 Announce Type: replace
Abstract: Proximal Policy Optimization algorithm employing a clipped surrogate objective (PPO-Clip) is a prominent exemplar of the policy optimization methods. However, despite its remarkable empirical success, PPO-Clip lacks theoretical substantiation to date. In this paper, we contribute to the field by establishing the first global convergence results of a PPO-Clip variant in both tabular and neural function approximation settings. Our findings highlight the $O(1/\sqrt{T})$ min-iterate convergence rate specifically in the context of neural function approximation. We …

abstract algorithm arxiv clip convergence cs.ai cs.lg global optimization paper policy ppo success type