Gaga: Group Any Gaussians via 3D-aware Memory Bank

arXiv:2404.07977v1 Announce Type: new
Abstract: We introduce Gaga, a framework that reconstructs and segments open-world 3D scenes by leveraging inconsistent 2D masks predicted by zero-shot segmentation models. Contrasted to prior 3D scene segmentation approaches that heavily rely on video object tracking, Gaga utilizes spatial information and effectively associates object masks across diverse camera poses. By eliminating the assumption of continuous view changes in training images, Gaga demonstrates robustness to variations in camera poses, particularly beneficial for sparsely sampled images, ensuring …

3d scenes abstract arxiv bank cs.cv diverse framework information masks memory object open-world prior segmentation spatial tracking type via video world zero-shot