Diverse Randomized Value Functions: A Provably Pessimistic Approach for Offline Reinforcement Learning

arXiv:2404.06188v1 Announce Type: new
Abstract: Offline Reinforcement Learning (RL) faces distributional shift and unreliable value estimation, especially for out-of-distribution (OOD) actions. To address this, existing uncertainty-based methods penalize the value function with uncertainty quantification and demand numerous ensemble networks, posing computational challenges and suboptimal outcomes. In this paper, we introduce a novel strategy employing diverse randomized value functions to estimate the posterior distribution of $Q$-values. It provides robust uncertainty quantification and estimates lower confidence bounds (LCB) of $Q$-values. By applying …

abstract arxiv challenges computational cs.ai cs.lg demand distribution diverse ensemble function functions networks offline paper quantification reinforcement reinforcement learning shift type uncertainty value