Function-space Parameterization of Neural Networks for Sequential Learning

arXiv:2403.10929v1 Announce Type: cross
Abstract: Sequential learning paradigms pose challenges for gradient-based deep learning due to difficulties incorporating new data and retaining prior knowledge. While Gaussian processes elegantly tackle these problems, they struggle with scalability and handling rich inputs, such as images. To address these issues, we introduce a technique that converts neural networks from weight space to function space, through a dual parameterization. Our parameterization offers: (i) a way to scale function-space methods to large data sets via sparsification, …

abstract arxiv challenges cs.lg data deep learning function gaussian processes gradient images inputs knowledge networks neural networks prior processes scalability space stat.ml struggle type