Open-Set 3D Semantic Instance Maps for Vision Language Navigation -- O3D-SIM

arXiv:2404.17922v1 Announce Type: new
Abstract: Humans excel at forming mental maps of their surroundings, equipping them to understand object relationships and navigate based on language queries. Our previous work SI Maps [1] showed that having instance-level information and the semantic understanding of an environment helps significantly improve performance for language-guided tasks. We extend this instance-level approach to 3D while increasing the pipeline's robustness and improving quantitative and qualitative results. Our method leverages foundational models for object recognition, image segmentation, and …

abstract arxiv cs.cv cs.ro environment excel humans information instance language maps navigation object performance queries relationships semantic set sim them type understanding vision work