Where Should I Spend My FLOPS? Efficiency Evaluations of Visual Pre-training Methods. (arXiv:2209.15589v3 [cs.CV] UPDATED)

Self-supervised methods have achieved remarkable success in transfer
learning, often achieving the same or better accuracy than supervised
pre-training. Most prior work has done so by increasing pre-training
computation by adding complex data augmentation, multiple views, or lengthy
training schedules. In this work, we investigate a related, but orthogonal
question: given a fixed FLOP budget, what are the best datasets, models, and
(self-)supervised training methods for obtaining high accuracy on
representative visual tasks? Given the availability of large datasets, this …

arxiv efficiency pre-training training