LangNav: Language as a Perceptual Representation for Navigation

arXiv:2310.07889v2 Announce Type: replace
Abstract: We explore the use of language as a perceptual representation for vision-and-language navigation (VLN), with a focus on low-data settings. Our approach uses off-the-shelf vision systems for image captioning and object detection to convert an agent's egocentric panoramic view at each time step into natural language descriptions. We then finetune a pretrained language model to select an action, based on the current view and the trajectory history, that would best fulfill the navigation instructions. In …

abstract agent arxiv captioning cs.ai cs.cl cs.cv cs.ro data detection explore focus image language low natural natural language navigation object representation systems type view vision vision-and-language