Depth Separation in Norm-Bounded Infinite-Width Neural Networks

arXiv:2402.08808v1 Announce Type: new
Abstract: We study depth separation in infinite-width neural networks, where complexity is controlled by the overall squared $\ell_2$-norm of the weights (sum of squares of all weights in the network). Whereas previous depth separation results focused on separation in terms of width, such results do not give insight into whether depth determines if it is possible to learn a network that generalizes well even when the network width is unbounded. Here, we study separation in terms …

abstract arxiv complexity cs.lg insight network networks neural networks norm squares stat.ml study terms type