PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain

arXiv:2402.15527v1 Announce Type: cross
Abstract: We present PCA-Bench, a multimodal decision-making benchmark for evaluating the integrated capabilities of Multimodal Large Language Models (MLLMs). Departing from previous benchmarks focusing on simplistic tasks and individual model capability, PCA-Bench introduces three complex scenarios: autonomous driving, domestic robotics, and open-world games. Given task instructions and diverse contexts, the model is required to seamlessly integrate multiple capabilities of Perception, Cognition, and Action in a reasoning chain to make accurate decisions. Moreover, PCA-Bench features error localization …

abstract arxiv autonomous autonomous driving benchmark benchmarks capabilities capability cognition cs.ai cs.cl cs.cv decision driving games language language models large language large language models making mllms multimodal open-world perception robotics tasks type world