The Effective Horizon Explains Deep RL Performance in Stochastic Environments

arXiv:2312.08369v2 Announce Type: replace-cross
Abstract: Reinforcement learning (RL) theory has largely focused on proving minimax sample complexity bounds. These require strategic exploration algorithms that use relatively limited function classes for representing the policy or value function. Our goal is to explain why deep RL algorithms often perform well in practice, despite using random exploration and much more expressive function classes like neural networks. Our work arrives at an explanation by showing that many stochastic MDPs can be solved by performing …

arxiv cs.ai cs.lg deep rl environments horizon performance stat.ml stochastic type