Track2Act: Predicting Point Tracks from Internet Videos enables Diverse Zero-shot Robot Manipulation

arXiv:2405.01527v1 Announce Type: cross
Abstract: We seek to learn a generalizable goal-conditioned policy that enables zero-shot robot manipulation: interacting with unseen objects in novel scenes without test-time adaptation. While typical approaches rely on a large amount of demonstration data for such generalization, we propose an approach that leverages web videos to predict plausible interaction plans and learns a task-agnostic transformation to obtain robot actions in the real world. Our framework,Track2Act predicts tracks of how points in an image should move …

arxiv cs.cv cs.ro diverse internet manipulation robot robot manipulation type videos zero-shot