Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Space

arXiv:2307.01177v2 Announce Type: replace
Abstract: To characterize the function space explored by neural networks (NNs) is an important aspect of learning theory. In this work, noticing that a multi-layer NN generates implicitly a hierarchy of reproducing kernel Hilbert spaces (RKHSs) - named a neural Hilbert ladder (NHL) - we define the function space as an infinite union of RKHSs, which generalizes the existing Barron space theory of two-layer NNs. We then establish several theoretical properties of the new space. First, …

abstract arxiv cs.lg function kernel layer math.fa math.oc math.pr networks neural networks nns space spaces stat.ml theory type work