Adaptive Tree Backup Algorithms for Temporal-Difference Reinforcement Learning. (arXiv:2206.01896v1 [cs.LG])

Q($\sigma$) is a recently proposed temporal-difference learning method that
interpolates between learning from expected backups and sampled backups. It has
been shown that intermediate values for the interpolation parameter $\sigma \in
[0,1]$ perform better in practice, and therefore it is commonly believed that
$\sigma$ functions as a bias-variance trade-off parameter to achieve these
improvements. In our work, we disprove this notion, showing that the choice of
$\sigma=0$ minimizes variance without increasing bias. This indicates that
$\sigma$ must have some other …

algorithms arxiv backup difference learning reinforcement reinforcement learning temporal tree