Controlling the Inductive Bias of Wide Neural Networks by Modifying the Kernel's Spectrum

arXiv:2307.14531v2 Announce Type: replace
Abstract: Wide neural networks are biased towards learning certain functions, influencing both the rate of convergence of gradient descent (GD) and the functions that are reachable with GD in finite training time. As such, there is a great need for methods that can modify this bias according to the task at hand. To that end, we introduce Modified Spectrum Kernels (MSKs), a novel family of constructed kernels that can be used to approximate kernels with desired …

abstract arxiv bias convergence cs.lg functions gradient inductive kernel networks neural networks rate spectrum training type