Colored Noise in PPO: Improved Exploration and Performance through Correlated Action Sampling

arXiv:2312.11091v2 Announce Type: replace
Abstract: Proximal Policy Optimization (PPO), a popular on-policy deep reinforcement learning method, employs a stochastic policy for exploration. In this paper, we propose a colored noise-based stochastic policy variant of PPO. Previous research highlighted the importance of temporal correlation in action noise for effective exploration in off-policy reinforcement learning. Building on this, we investigate whether correlated noise can also enhance exploration in on-policy methods like PPO. We discovered that correlated noise for action selection improves learning …

abstract action arxiv correlation cs.lg exploration importance noise optimization paper performance policy popular ppo reinforcement reinforcement learning replace research sampling stochastic temporal through type