3D Open-Vocabulary Panoptic Segmentation with 2D-3D Vision-Language Distillation

arXiv:2401.02402v3 Announce Type: replace
Abstract: 3D panoptic segmentation is a challenging perception task, especially in autonomous driving. It aims to predict both semantic and instance annotations for 3D points in a scene. Although prior 3D panoptic segmentation approaches have achieved great performance on closed-set benchmarks, generalizing these approaches to unseen things and unseen stuff categories remains an open problem. For unseen object categories, 2D open-vocabulary segmentation has achieved promising results that solely rely on frozen CLIP backbones and ensembling multiple …

abstract annotations arxiv autonomous autonomous driving benchmarks cs.cv distillation driving instance language panoptic segmentation perception performance prior segmentation semantic set type vision