LOC-ZSON: Language-driven Object-Centric Zero-Shot Object Retrieval and Navigation

arXiv:2405.05363v1 Announce Type: new
Abstract: In this paper, we present LOC-ZSON, a novel Language-driven Object-Centric image representation for object navigation task within complex scenes. We propose an object-centric image representation and corresponding losses for visual-language model (VLM) fine-tuning, which can handle complex object-level queries. In addition, we design a novel LLM-based augmentation and prompt templates for stability during training and zero-shot inference. We implement our method on Astro robot and deploy it in both simulated and real-world environments for zero-shot …

abstract arxiv augmentation cs.cv cs.ro design fine-tuning image language language model llm loc losses navigation novel object paper queries representation retrieval type visual vlm zero-shot