Predicting the Performance of Foundation Models via Agreement-on-the-Line

arXiv:2404.01542v1 Announce Type: new
Abstract: Estimating the out-of-distribution performance in regimes where labels are scarce is critical to safely deploy foundation models. Recently, it was shown that ensembles of neural networks observe the phenomena ``agreement-on-the-line'', which can be leveraged to reliably predict OOD performance without labels. However, in contrast to classical neural networks that are trained on in-distribution data from scratch for numerous epochs, foundation models undergo minimal finetuning from heavily pretrained weights, which may reduce the ensemble diversity needed …

abstract agreement arxiv contrast cs.lg deploy distribution foundation however labels line networks neural networks observe performance type via