ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understanding

arXiv:2305.08275v3 Announce Type: replace
Abstract: Recent advancements in multimodal pre-training have shown promising efficacy in 3D representation learning by aligning multimodal features across 3D shapes, their 2D counterparts, and language descriptions. However, the methods used by existing frameworks to curate such multimodal data, in particular language descriptions for 3D shapes, are not scalable, and the collected language descriptions are not diverse. To address this, we introduce ULIP-2, a simple yet effective tri-modal pre-training framework that leverages large multimodal models to …

arxiv cs.cv multimodal pre-training scalable training type understanding