[R] Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model

**arXiv**: [https://arxiv.org/abs/2310.17653](https://arxiv.org/abs/2310.17653)

**OpenReview**: [https://openreview.net/forum?id=m50eKHCttz](https://openreview.net/forum?id=m50eKHCttz)

**Abstract**:

>Training deep networks requires various design decisions regarding for instance their architecture, data augmentation, or optimization. In this work, we find these training variations to result in networks learning unique feature sets from the data. Using public model libraries comprising thousands of models trained on canonical datasets like ImageNet, we observe that for arbitrary pairings of pretrained models, one model extracts significant data context unavailable in the other -- independent of overall performance. Given any …

abstract architecture augmentation canonical data datasets decisions design feature imagenet instance libraries machinelearning networks observe optimization pretrained models public training work