Probing Multimodal LLMs as World Models for Driving

arXiv:2405.05956v1 Announce Type: cross
Abstract: We provide a sober look at the application of Multimodal Large Language Models (MLLMs) within the domain of autonomous driving and challenge/verify some common assumptions, focusing on their ability to reason and interpret dynamic driving scenarios through sequences of images/frames in a closed-loop control environment. Despite the significant advancements in MLLMs like GPT-4V, their performance in complex, dynamic driving environments remains largely untested and presents a wide area of exploration. We conduct a comprehensive experimental …

abstract application arxiv assumptions autonomous autonomous driving challenge control cs.cv cs.ro domain driving dynamic environment images language language models large language large language models llms look loop mllms multimodal multimodal llms reason sober through type verify world world models