Sieve: Multimodal Dataset Pruning Using Image Captioning Models

arXiv:2310.02110v2 Announce Type: replace
Abstract: Vision-Language Models (VLMs) are pretrained on large, diverse, and noisy web-crawled datasets. This underscores the critical need for dataset pruning, as the quality of these datasets is strongly correlated with the performance of VLMs on downstream tasks. Using CLIPScore from a pretrained model to only train models using highly-aligned samples is one of the most successful methods for pruning. We argue that this approach suffers from multiple limitations including: false positives and negatives due to …

abstract arxiv captioning cs.cv dataset datasets diverse image language language models multimodal performance pruning quality tasks train type vision vision-language models vlms web