Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splatting

arXiv:2403.15624v1 Announce Type: new
Abstract: Open-vocabulary 3D scene understanding presents a significant challenge in computer vision, withwide-ranging applications in embodied agents and augmented reality systems. Previous approaches haveadopted Neural Radiance Fields (NeRFs) to analyze 3D scenes. In this paper, we introduce SemanticGaussians, a novel open-vocabulary scene understanding approach based on 3D Gaussian Splatting. Our keyidea is distilling pre-trained 2D semantics into 3D Gaussians. We design a versatile projection approachthat maps various 2Dsemantic features from pre-trained image encoders into a novel …

3d scenes abstract agents analyze applications arxiv augmented reality challenge computer computer vision cs.cv embodied fields neural radiance fields novel paper reality semantic systems type understanding vision