NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation

arXiv:2402.15852v1 Announce Type: new
Abstract: Vision-and-Language Navigation (VLN) stands as a key research problem of Embodied AI, aiming at enabling agents to navigate in unseen environments following linguistic instructions. In this field, generalization is a long-standing challenge, either to out-of-distribution scenes or from Sim to Real. In this paper, we propose NaVid, a video-based large vision language model (VLM), to mitigate such a generalization gap. NaVid makes the first endeavour to showcase the capability of VLMs to achieve state-of-the-art level …

abstract agents arxiv challenge cs.cv cs.ro distribution embodied embodied ai enabling environments key language navigation next paper research sim type video vision vision-and-language vlm