DNAct: Diffusion Guided Multi-Task 3D Policy Learning

arXiv:2403.04115v1 Announce Type: cross
Abstract: This paper presents DNAct, a language-conditioned multi-task policy framework that integrates neural rendering pre-training and diffusion training to enforce multi-modality learning in action sequence spaces. To learn a generalizable multi-task policy with few demonstrations, the pre-training phase of DNAct leverages neural rendering to distill 2D semantic features from foundation models such as Stable Diffusion to a 3D space, which provides a comprehensive semantic understanding regarding the scene. Consequently, it allows various applications to challenging robotic …

abstract arxiv cs.ai cs.cv cs.ro diffusion features framework language learn neural rendering paper policy pre-training rendering semantic spaces training type