No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance

arXiv:2404.04125v1 Announce Type: cross
Abstract: Web-crawled pretraining datasets underlie the impressive "zero-shot" evaluation performance of multimodal models, such as CLIP for classification/retrieval and Stable-Diffusion for image generation. However, it is unclear how meaningful the notion of "zero-shot" generalization is for such multimodal models, as it is not known to what extent their pretraining datasets encompass the downstream concepts targeted for during "zero-shot" evaluation. In this work, we ask: How is the performance of multimodal models on downstream concepts influenced by …

abstract arxiv classification clip concept cs.cl cs.cv cs.lg data datasets diffusion evaluation however image image generation multimodal multimodal model multimodal models notion performance pretraining retrieval type web zero-shot