On Gap-dependent Bounds for Offline Reinforcement Learning. (arXiv:2206.00177v2 [cs.LG] UPDATED)

This paper presents a systematic study on gap-dependent sample complexity in
offline reinforcement learning. Prior work showed when the density ratio
between an optimal policy and the behavior policy is upper bounded (the optimal
policy coverage assumption), then the agent can achieve an
$O\left(\frac{1}{\epsilon^2}\right)$ rate, which is also minimax optimal. We
show under the optimal policy coverage assumption, the rate can be improved to
$O\left(\frac{1}{\epsilon}\right)$ when there is a positive sub-optimality gap
in the optimal $Q$-function. Furthermore, we show when …

arxiv gap learning lg reinforcement reinforcement learning