MileBench: Benchmarking MLLMs in Long Context

arXiv:2404.18532v1 Announce Type: cross
Abstract: Despite the advancements and impressive performance of Multimodal Large Language Models (MLLMs) on benchmarks, their effectiveness in real-world, long-context, and multi-image tasks is unclear due to the benchmarks' limited scope. Existing benchmarks often focus on single-image and short-text samples, and when assessing multi-image tasks, they either limit the image count or focus on specific task (e.g time-series captioning), potentially obscuring the performance challenges of MLLMs. To address these limitations, we introduce MileBench, a pioneering benchmark …

abstract arxiv benchmarking benchmarks context cs.ai cs.cl cs.cv cs.lg focus image language language models large language large language models mllms multimodal performance samples tasks text type world