Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization. (arXiv:2212.10445v3 [cs.LG] UPDATED)

Foundation models are redefining how AI systems are built. Practitioners now
follow a standard procedure to build their machine learning solutions: from a
pre-trained foundation model, they fine-tune the weights on the target task of
interest. So, the Internet is swarmed by a handful of foundation models
fine-tuned on many diverse tasks: these individual fine-tunings exist in
isolation without benefiting from each other. In our opinion, this is a missed
opportunity, as these specialized models contain rich and diverse features. …

ai systems arxiv build distribution diverse foundation foundation model internet machine machine learning recycling solutions standard systems