Input-gradient space particle inference for neural network ensembles

arXiv:2306.02775v2 Announce Type: replace-cross
Abstract: Deep Ensembles (DEs) demonstrate improved accuracy, calibration and robustness to perturbations over single neural networks partly due to their functional diversity. Particle-based variational inference (ParVI) methods enhance diversity by formalizing a repulsion term based on a network similarity kernel. However, weight-space repulsion is inefficient due to over-parameterization, while direct function-space repulsion has been found to produce little improvement over DEs. To sidestep these difficulties, we propose First-order Repulsive Deep Ensemble (FoRDE), an ensemble learning method …

abstract accuracy arxiv cs.lg des diversity functional gradient inference kernel network networks neural network neural networks robustness space stat.ml type