Dissecting Deep RL with High Update Ratios: Combatting Value Overestimation and Divergence

arXiv:2403.05996v1 Announce Type: new
Abstract: We show that deep reinforcement learning can maintain its ability to learn without resetting network parameters in settings where the number of gradient updates greatly exceeds the number of environment samples. Under such large update-to-data ratios, a recent study by Nikishin et al. (2022) suggested the emergence of a primacy bias, in which agents overfit early interactions and downplay later experience, impairing their ability to learn. In this work, we dissect the phenomena underlying the …

abstract arxiv cs.ai cs.lg data deep rl divergence environment gradient learn network parameters reinforcement reinforcement learning samples show study type update updates value