An Empirical Study of Implicit Regularization in Deep Offline RL. (arXiv:2207.02099v1 [cs.LG])

Deep neural networks are the most commonly used function approximators in
offline Reinforcement Learning these days. Prior works have shown that neural
nets trained with TD-learning and gradient descent can exhibit implicit
regularization that can be characterized by under-parameterization of these
networks. Specifically, the rank of the penultimate feature layer, also called
\textit{effective rank}, has been observed to drastically collapse during the
training. In turn, this collapse has been argued to reduce the model's ability
to further adapt in later …

arxiv lg regularization rl study