Think-Program-reCtify: 3D Situated Reasoning with Large Language Models

arXiv:2404.14705v1 Announce Type: new
Abstract: This work addresses the 3D situated reasoning task which aims to answer questions given egocentric observations in a 3D environment. The task remains challenging as it requires comprehensive 3D perception and complex reasoning skills. End-to-end models trained on supervised data for 3D situated reasoning suffer from data scarcity and generalization ability. Inspired by the recent success of leveraging large language models (LLMs) for visual reasoning, we propose LLM-TPC, a novel framework that leverages the planning, …

arxiv cs.cv language language models large language large language models reasoning think type