Using Left and Right Brains Together: Towards Vision and Language Planning

arXiv:2402.10534v1 Announce Type: new
Abstract: Large Language Models (LLMs) and Large Multi-modality Models (LMMs) have demonstrated remarkable decision masking capabilities on a variety of tasks. However, they inherently operate planning within the language space, lacking the vision and spatial imagination ability. In contrast, humans utilize both left and right hemispheres of the brain for language and visual planning during the thinking process. Therefore, we introduce a novel vision-language planning framework in this work to perform concurrent visual and language planning …

abstract arxiv brains capabilities contrast cs.cv decision humans imagination language language models large language large language models llms lmms masking planning space spatial tasks together type vision