Unified Scene Representation and Reconstruction for 3D Large Language Models

arXiv:2404.13044v1 Announce Type: new
Abstract: Enabling Large Language Models (LLMs) to interact with 3D environments is challenging. Existing approaches extract point clouds either from ground truth (GT) geometry or 3D scenes reconstructed by auxiliary models. Text-image aligned 2D features from CLIP are then lifted to point clouds, which serve as inputs for LLMs. However, this solution lacks the establishment of 3D point-to-point connections, leading to a deficiency of spatial structure information. Concurrently, the absence of integration and unification between the …

3d scenes abstract arxiv clip cs.cv enabling environments extract features geometry image language language models large language large language models llms representation serve text text-image truth type