Composing Pre-Trained Object-Centric Representations for Robotics From "What" and "Where" Foundation Models

arXiv:2404.13474v1 Announce Type: cross
Abstract: There have recently been large advances both in pre-training visual representations for robotic control and segmenting unknown category objects in general images. To leverage these for improved robot learning, we propose $\textbf{POCR}$, a new framework for building pre-trained object-centric representations for robotic control. Building on theories of "what-where" representations in psychology and computer vision, we use segmentations from a pre-trained model to stably locate across timesteps, various entities in the scene, capturing "where" information. To …

abstract advances arxiv building control cs.ai cs.cv cs.lg cs.ro foundation framework general images object objects pre-training robot robotic robotics training type visual