EgoPlan-Bench: Benchmarking Egocentric Embodied Planning with Multimodal Large Language Models

arXiv:2312.06722v2 Announce Type: replace
Abstract: Multimodal Large Language Models, combining the remarkable reasoning and generalization capabilities of Large Language Models (LLMs) with the ability to comprehend visual inputs, have opened up new avenues for embodied task planning. Given diverse environmental inputs, including real-time task progress, visual observations, and open-form language instructions, a proficient task planner is expected to predict feasible actions, which is a feat inherently achievable by Multimodal Large Language Models (MLLMs). In this paper, we aim to quantitatively …

abstract arxiv benchmarking capabilities cs.cl cs.cv cs.ro diverse embodied environmental form inputs language language models large language large language models llms multimodal planning progress real-time reasoning type visual