DINOBot: Robot Manipulation via Retrieval and Alignment with Vision Foundation Models

arXiv:2402.13181v1 Announce Type: cross
Abstract: We propose DINOBot, a novel imitation learning framework for robot manipulation, which leverages the image-level and pixel-level capabilities of features extracted from Vision Transformers trained with DINO. When interacting with a novel object, DINOBot first uses these features to retrieve the most visually similar object experienced during human demonstrations, and then uses this object to align its end-effector with the novel object to enable effective interaction. Through a series of real-world experiments on everyday tasks, …

alignment arxiv cs.lg cs.ro foundation manipulation retrieval robot robot manipulation type via vision