Towards Principled Representation Learning from Videos for Reinforcement Learning

arXiv:2403.13765v1 Announce Type: new
Abstract: We study pre-training representations for decision-making using video data, which is abundantly available for tasks such as game agents and software testing. Even though significant empirical advances have been made on this problem, a theoretical understanding remains absent. We initiate the theoretical investigation into principled approaches for representation learning and focus on learning the latent state representations of the underlying MDP using video data. We study two types of settings: one where there is iid …

abstract advances agents arxiv cs.ai cs.cv cs.lg data decision game investigation making pre-training reinforcement reinforcement learning representation representation learning software software testing study tasks testing training type understanding video video data videos