Embodied Understanding of Driving Scenarios

arXiv:2403.04593v1 Announce Type: new
Abstract: Embodied scene understanding serves as the cornerstone for autonomous agents to perceive, interpret, and respond to open driving scenarios. Such understanding is typically founded upon Vision-Language Models (VLMs). Nevertheless, existing VLMs are restricted to the 2D domain, devoid of spatial awareness and long-horizon extrapolation proficiencies. We revisit the key aspects of autonomous driving and formulate appropriate rubrics. Hereby, we introduce the Embodied Language Model (ELM), a comprehensive framework tailored for agents' understanding of driving scenes …

abstract agents arxiv autonomous autonomous agents cs.cv domain driving embodied horizon key language language models spatial the key type understanding vision vision-language models vlms