Less is More: Data Value Estimation for Visual Instruction Tuning

arXiv:2403.09559v1 Announce Type: cross
Abstract: Visual instruction tuning is the key to building multimodal large language models (MLLMs), which greatly improves the reasoning capabilities of large language models (LLMs) in vision scenario. However, existing MLLMs mostly rely on a mixture of multiple highly diverse visual instruction datasets for training (even more than a million instructions), which may introduce data redundancy. To investigate this issue, we conduct a series of empirical studies, which reveal a significant redundancy within the visual instruction …

abstract arxiv building capabilities cs.cl cs.cv data datasets data value diverse however key language language models large language large language models llms mllms multimodal multiple reasoning the key training type value vision visual