DiffCLIP: Leveraging Stable Diffusion for Language Grounded 3D Classification

arXiv:2305.15957v3 Announce Type: replace
Abstract: Large pre-trained models have had a significant impact on computer vision by enabling multi-modal learning, where the CLIP model has achieved impressive results in image classification, object detection, and semantic segmentation. However, the model's performance on 3D point cloud processing tasks is limited due to the domain gap between depth maps from 3D projection and training images of CLIP. This paper proposes DiffCLIP, a new pre-training framework that incorporates stable diffusion with ControlNet to minimize …

abstract arxiv classification clip cloud computer computer vision cs.cv detection diffusion enabling however image impact language modal multi-modal object performance pre-trained models processing results segmentation semantic s performance stable diffusion tasks type vision