Learning and Leveraging World Models in Visual Representation Learning

arXiv:2403.00504v1 Announce Type: cross
Abstract: Joint-Embedding Predictive Architecture (JEPA) has emerged as a promising self-supervised approach that learns by leveraging a world model. While previously limited to predicting missing parts of an input, we explore how to generalize the JEPA prediction task to a broader set of corruptions. We introduce Image World Models, an approach that goes beyond masked image modeling and learns to predict the effect of global photometric transformations in latent space. We study the recipe of learning …

abstract architecture arxiv cs.ai cs.cv cs.lg embedding explore image jepa prediction predictive representation representation learning set type visual world world models