Vision-Language Models as a Source of Rewards

arXiv:2312.09187v2 Announce Type: replace
Abstract: Building generalist agents that can accomplish many goals in rich open-ended environments is one of the research frontiers for reinforcement learning. A key limiting factor for building generalist agents with RL has been the need for a large number of reward functions for achieving different goals. We investigate the feasibility of using off-the-shelf vision-language models, or VLMs, as sources of rewards for reinforcement learning agents. We show how rewards for visual achievement of a variety …

abstract agents arxiv building cs.lg environments frontiers functions key language language models reinforcement reinforcement learning research type vision vision-language models