FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understanding

arXiv:2401.01970v2 Announce Type: replace
Abstract: Precisely perceiving the geometric and semantic properties of real-world 3D objects is crucial for the continued evolution of augmented reality and robotic applications. To this end, we present Foundation Model Embedded Gaussian Splatting (FMGS), which incorporates vision-language embeddings of foundation models into 3D Gaussian Splatting (GS). The key contribution of this work is an efficient method to reconstruct and represent 3D vision-language models. This is achieved by distilling feature maps generated from image-based foundation models …

3d objects abstract applications arxiv augmented reality cs.ai cs.cv embedded embeddings evolution foundation foundation model language objects reality robotic semantic type understanding vision vision-language world