Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficulty. (arXiv:2209.09658v1 [cs.LG])

Among attempts at giving a theoretical account of the success of deep neural
networks, a recent line of work has identified a so-called `lazy' regime in
which the network can be well approximated by its linearization around
initialization. Here we investigate the comparative effect of the lazy (linear)
and feature learning (non-linear) regimes on subgroups of examples based on
their difficulty. Specifically, we show that easier examples are given more
weight in feature learning mode, resulting in faster training compared …

arxiv example impacts lazy linearization networks