An Embodied Generalist Agent in 3D World

arXiv:2311.12871v2 Announce Type: replace-cross
Abstract: Leveraging massive knowledge and learning schemes from large language models (LLMs), recent machine learning models show notable successes in building generalist agents that exhibit the capability of general-purpose task solving in diverse domains, including natural language processing, computer vision, and robotics. However, a significant challenge remains as these models exhibit limited ability in understanding and interacting with the 3D world. We argue this limitation significantly hinders the current models from performing real-world tasks and further …

abstract agent agents arxiv building capability challenge computer computer vision cs.ai cs.cl cs.cv cs.lg diverse domains embodied general however knowledge language language models language processing large language large language models llms machine machine learning machine learning models massive natural natural language natural language processing processing robotics show type vision world