On the Estimation Bias in Double Q-Learning. (arXiv:2109.14419v3 [cs.LG] UPDATED)

Double Q-learning is a classical method for reducing overestimation bias,
which is caused by taking maximum estimated values in the Bellman operation.
Its variants in the deep Q-learning paradigm have shown great promise in
producing reliable value prediction and improving learning performance.
However, as shown by prior work, double Q-learning is not fully unbiased and
suffers from underestimation bias. In this paper, we show that such
underestimation bias may lead to multiple non-optimal fixed points under an
approximate Bellman operator. …

arxiv bias learning