Learning by Watching: A Review of Video-based Learning Approaches for Robot Manipulation

Robot learning of manipulation skills is hindered by the scarcity of diverse, unbiased datasets. While curated datasets can help, challenges remain in generalizability and real-world transfer. Meanwhile, large-scale "in-the-wild" video datasets have driven progress in computer vision through self-supervised techniques. Translating this to robotics, recent works have explored learning manipulation skills by passively watching abundant videos sourced online. Showing promising results, such video-based learning paradigms provide scalable supervision while reducing dataset bias. This survey reviews foundations such as video feature …

challenges computer computer vision cs.ai cs.cv cs.lg cs.ro datasets diverse manipulation progress review robot robotics robot manipulation scale skills through transfer unbiased video vision world