Duoduo CLIP: Efficient 3D Understanding with Multi-View Images

arXiv:2406.11579v1 Announce Type: new
Abstract: We introduce Duoduo CLIP, a model for 3D representation learning that learns shape encodings from multi-view images instead of point-clouds. The choice of multi-view images allows us to leverage 2D priors from off-the-shelf CLIP models to facilitate fine-tuning with 3D data. Our approach not only shows better generalization compared to existing point cloud methods, but also reduces GPU requirements and training time. In addition, we modify the model with cross-view attention to leverage information across …

abstract arxiv clip cs.cv data fine-tuning images off representation representation learning shape shows tuning type understanding view