Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. (arXiv:2203.05482v2 [cs.LG] UPDATED)

The conventional recipe for maximizing model accuracy is to (1) train
multiple models with various hyperparameters and (2) pick the individual model
which performs best on a held-out validation set, discarding the remainder. In
this paper, we revisit the second step of this procedure in the context of
fine-tuning large pre-trained models, where fine-tuned models often appear to
lie in a single low error basin. We show that averaging the weights of multiple
models fine-tuned with different hyperparameter configurations often …

accuracy arxiv inference lg time