Stop Regressing: Training Value Functions via Classification for Scalable Deep RL

arXiv:2403.03950v1 Announce Type: new
Abstract: Value functions are a central component of deep reinforcement learning (RL). These functions, parameterized by neural networks, are trained using a mean squared error regression objective to match bootstrapped target values. However, scaling value-based RL methods that use regression to large networks, such as high-capacity Transformers, has proven challenging. This difficulty is in stark contrast to supervised learning: by leveraging a cross-entropy classification loss, supervised methods have scaled reliably to massive networks. Observing this discrepancy, …

abstract arxiv classification cs.ai cs.lg deep rl error functions however match mean networks neural networks regression reinforcement reinforcement learning scalable scaling stat.ml training type value values via