Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models

arXiv:2403.00231v1 Announce Type: new
Abstract: Large vision-language models (LVLMs), exemplified by GPT-4V, excel across diverse tasks involving concrete images from natural scenes. However, their ability to interpret abstract figures, such as geometry shapes and scientific plots, remains limited due to a scarcity of training datasets in scientific domains. To fill this gap, we introduce Multimodal ArXiv, consisting of ArXivCap and ArXivQA, for enhancing LVLMs scientific comprehension. ArXivCap is a figure-caption dataset comprising 6.4M images and 3.9M captions sourced from 572K …

abstract arxiv concrete cs.cl cs.cv dataset datasets diverse domains excel geometry gpt gpt-4v images language language models multimodal natural plots tasks training training datasets type vision vision-language models