Large Language Models as Generalizable Policies for Embodied Tasks

arXiv:2310.17722v2 Announce Type: replace
Abstract: We show that large language models (LLMs) can be adapted to be generalizable policies for embodied visual tasks. Our approach, called Large LAnguage model Reinforcement Learning Policy (LLaRP), adapts a pre-trained frozen LLM to take as input text instructions and visual egocentric observations and output actions directly in the environment. Using reinforcement learning, we train LLaRP to see and act solely through environmental interactions. We show that LLaRP is robust to complex paraphrasings of task …

arxiv cs.ai cs.cl cs.lg embodied language language models large language large language models policies tasks type