Scaling (Down) CLIP: A Comprehensive Analysis of Data, Architecture, and Training Strategies

arXiv:2404.08197v1 Announce Type: new
Abstract: This paper investigates the performance of the Contrastive Language-Image Pre-training (CLIP) when scaled down to limited computation budgets. We explore CLIP along three dimensions: data, architecture, and training strategies. With regards to data, we demonstrate the significance of high-quality training data and show that a smaller dataset of high-quality data can outperform a larger dataset with lower quality. We also examine how model performance varies with different dataset sizes, suggesting that smaller ViT models are …

abstract analysis architecture arxiv budgets clip computation cs.cv data dimensions explore image language paper performance pre-training quality scaling show significance strategies training training data type