Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Control

arXiv:2309.14597v3 Announce Type: replace
Abstract: Deep reinforcement learning agents for continuous control are known to exhibit significant instability in their performance over time. In this work, we provide a fresh perspective on these behaviors by studying the return landscape: the mapping between a policy and a return. We find that popular algorithms traverse noisy neighborhoods of this landscape, in which a single update to the policy parameters leads to a wide range of returns. By taking a distributional view of …

abstract agents arxiv continuous control cs.lg landscape mapping optimization performance perspective policy reinforcement reinforcement learning studying type work