An Empirical Study of Implicit Regularization in Deep Offline RL. (arXiv:2207.02099v2 [cs.LG] UPDATED)

Deep neural networks are the most commonly used function approximators in
offline reinforcement learning. Prior works have shown that neural nets trained
with TD-learning and gradient descent can exhibit implicit regularization that
can be characterized by under-parameterization of these networks. Specifically,
the rank of the penultimate feature layer, also called \textit{effective rank},
has been observed to drastically collapse during the training. In turn, this
collapse has been argued to reduce the model's ability to further adapt in
later stages of …

arxiv lg regularization rl study