Rethinking 3D Dense Caption and Visual Grounding in A Unified Framework through Prompt-based Localization

arXiv:2404.11064v1 Announce Type: new
Abstract: 3D Visual Grounding (3DVG) and 3D Dense Captioning (3DDC) are two crucial tasks in various 3D applications, which require both shared and complementary information in localization and visual-language relationships. Therefore, existing approaches adopt the two-stage "detect-then-describe/discriminate" pipeline, which relies heavily on the performance of the detector, resulting in suboptimal performance. Inspired by DETR, we propose a unified framework, 3DGCTR, to jointly solve these two distinct but closely related tasks in an end-to-end fashion. The key …

abstract applications arxiv captioning cs.ai cs.cv framework information language localization pipeline prompt relationships stage tasks through type visual