Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learning

In this study, we explore the influence of different observation spaces on robot learning, focusing on three predominant modalities: RGB, RGB-D, and point cloud. Through extensive experimentation on over 17 varied contact-rich manipulation tasks, conducted across two benchmarks and simulators, we have observed a notable trend: point cloud-based methods, even those with the simplest designs, frequently surpass their RGB and RGB-D counterparts in performance. This remains consistent in both scenarios: training from scratch and utilizing pretraining. Furthermore, our findings indicate …

benchmarks cloud cs.ai cs.cv cs.lg cs.ro experimentation explore impact influence manipulation observation rgb-d robot spaces study tasks through trend