Pre-trained Text-to-Image Diffusion Models Are Versatile Representation Learners for Control

arXiv:2405.05852v1 Announce Type: cross
Abstract: Embodied AI agents require a fine-grained understanding of the physical world mediated through visual and language inputs. Such capabilities are difficult to learn solely from task-specific data. This has led to the emergence of pre-trained vision-language models as a tool for transferring representations learned from internet-scale data to downstream tasks and new domains. However, commonly used contrastively trained representations such as in CLIP have been shown to fail at enabling embodied agents to gain a …

abstract agents ai agents arxiv capabilities control cs.ai cs.cv cs.lg cs.ro data diffusion diffusion models embodied embodied ai emergence fine-grained image image diffusion inputs language language models learn representation stat.ml text text-to-image through tool type understanding vision vision-language vision-language models visual world