Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic

arXiv:2306.02865v4 Announce Type: replace
Abstract: Learning high-quality Q-value functions plays a key role in the success of many modern off-policy deep reinforcement learning (RL) algorithms. Previous works focus on addressing the value overestimation issue, an outcome of adopting function approximators and off-policy learning. Deviating from the common viewpoint, we observe that Q-values are indeed underestimated in the latter stage of the RL training process, primarily related to the use of inferior actions from the current policy in Bellman updates as …

abstract actor actor-critic algorithms arxiv cs.ai cs.lg focus function functions issue key modern policy quality reinforcement reinforcement learning role success type value