Learning Dynamics and Generalization in Reinforcement Learning. (arXiv:2206.02126v1 [cs.LG])

Solving a reinforcement learning (RL) problem poses two competing challenges:
fitting a potentially discontinuous value function, and generalizing well to
new observations. In this paper, we analyze the learning dynamics of temporal
difference algorithms to gain novel insight into the tension between these two
objectives. We show theoretically that temporal difference learning encourages
agents to fit non-smooth components of the value function early in training,
and at the same time induces the second-order effect of discouraging
generalization. We corroborate these …

arxiv dynamics learning reinforcement reinforcement learning