Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld

arXiv:2311.16714v2 Announce Type: replace
Abstract: While large language models (LLMs) excel in a simulated world of texts, they struggle to interact with the more realistic world without perceptions of other modalities such as visual or audio signals. Although vision-language models (VLMs) integrate LLM modules (1) aligned with static image features, and (2) may possess prior knowledge of world dynamics (as demonstrated in the text world), they have not been trained in an embodied visual world and thus cannot align with …

abstract agent arxiv audio cs.cv embodied excel image language language models large language large language models llm llms modal modules multi-modal struggle type vision vision-language models visual vlms world