Separation of scales and a thermodynamic description of feature learning in some CNNs. (arXiv:2112.15383v1 [stat.ML])

Deep neural networks (DNNs) are powerful tools for compressing and distilling
information. Due to their scale and complexity, often involving billions of
inter-dependent internal degrees of freedom, exact analysis approaches often
fall short. A common strategy in such cases is to identify slow degrees of
freedom that average out the erratic behavior of the underlying fast
microscopic variables. Here, we identify such a separation of scales occurring
in over-parameterized deep convolutional neural networks (CNNs) at the end of
training. It …

arxiv cnns learning ml