Sequence Compression Speeds Up Credit Assignment in Reinforcement Learning

arXiv:2405.03878v1 Announce Type: new
Abstract: Temporal credit assignment in reinforcement learning is challenging due to delayed and stochastic outcomes. Monte Carlo targets can bridge long delays between action and consequence but lead to high-variance targets due to stochasticity. Temporal difference (TD) learning uses bootstrapping to overcome variance but introduces a bias that can only be corrected through many iterations. TD($\lambda$) provides a mechanism to navigate this bias-variance tradeoff smoothly. Appropriately selecting $\lambda$ can significantly improve performance. Here, we propose Chunked-TD, …

abstract arxiv bias bootstrapping bridge compression credit cs.lg difference reinforcement reinforcement learning stochastic targets temporal type variance