The dynamics of representation learning in shallow, non-linear autoencoders. (arXiv:2201.02115v2 [stat.ML] UPDATED)

Autoencoders are the simplest neural network for unsupervised learning, and
thus an ideal framework for studying feature learning. While a detailed
understanding of the dynamics of linear autoencoders has recently been
obtained, the study of non-linear autoencoders has been hindered by the
technical difficulty of handling training data with non-trivial correlations -
a fundamental prerequisite for feature extraction. Here, we study the dynamics
of feature learning in non-linear, shallow autoencoders. We derive a set of
asymptotically exact equations that describe …

arxiv dynamics learning linear ml non-linear representation representation learning