A Note on Loss Functions and Error Compounding in Model-based Reinforcement Learning

arXiv:2404.09946v1 Announce Type: new
Abstract: This note clarifies some confusions (and perhaps throws out more) around model-based reinforcement learning and their theoretical understanding in the context of deep RL. Main topics of discussion are (1) how to reconcile model-based RL's bad empirical reputation on error compounding with its superior theoretical properties, and (2) the limitations of empirically popular losses. For the latter, concrete counterexamples for the "MuZero loss" are constructed to show that it not only fails in stochastic environments, …

abstract arxiv context cs.ai cs.lg deep rl error functions loss reinforcement reinforcement learning stat.ml topics type understanding