Half-Space Feature Learning in Neural Networks

arXiv:2404.04312v1 Announce Type: new
Abstract: There currently exist two extreme viewpoints for neural network feature learning -- (i) Neural networks simply implement a kernel method (a la NTK) and hence no features are learned (ii) Neural networks can represent (and hence learn) intricate hierarchical features suitable for the data. We argue in this paper neither interpretation is likely to be correct based on a novel viewpoint. Neural networks can be viewed as a mixture of experts, where each expert corresponds …

abstract arxiv cs.ai cs.lg cs.ne data feature features hierarchical kernel learn network networks neural network neural networks paper space type