ImageNot: A contrast with ImageNet preserves model rankings

arXiv:2404.02112v1 Announce Type: new
Abstract: We introduce ImageNot, a dataset designed to match the scale of ImageNet while differing drastically in other aspects. We show that key model architectures developed for ImageNet over the years rank identically when trained and evaluated on ImageNot to how they rank on ImageNet. This is true when training models from scratch or fine-tuning them. Moreover, the relative improvements of each model over earlier models strongly correlate in both datasets. We further give evidence that …

abstract architectures arxiv contrast cs.cv cs.lg dataset imagenet key match rankings scale show true type