Regularized Q-learning through Robust Averaging

arXiv:2405.02201v1 Announce Type: cross
Abstract: We propose a new Q-learning variant, called 2RA Q-learning, that addresses some weaknesses of existing Q-learning methods in a principled manner. One such weakness is an underlying estimation bias which cannot be controlled and often results in poor performance. We propose a distributionally robust estimator for the maximum expected value term, which allows us to precisely control the level of estimation bias introduced. The distributionally robust estimator admits a closed-form solution such that the proposed …

abstract arxiv bias cs.lg estimator math.oc maximum performance q-learning results robust through type value