Mildly Conservative Q-Learning for Offline Reinforcement Learning

arXiv:2206.04745v3 Announce Type: replace
Abstract: Offline reinforcement learning (RL) defines the task of learning from a static logged dataset without continually interacting with the environment. The distribution shift between the learned policy and the behavior policy makes it necessary for the value function to stay conservative such that out-of-distribution (OOD) actions will not be severely overestimated. However, existing approaches, penalizing the unseen actions or regularizing with the behavior policy, are too pessimistic, which suppresses the generalization of the value function …

arxiv cs.ai cs.lg offline q-learning reinforcement reinforcement learning type