Video as the New Language for Real-World Decision Making

arXiv:2402.17139v1 Announce Type: new
Abstract: Both text and video data are abundant on the internet and support large-scale self-supervised learning through next token or frame prediction. However, they have not been equally leveraged: language models have had significant real-world impact, whereas video generation has remained largely limited to media entertainment. Yet video data captures important information about the physical world that is difficult to express in language. To address this gap, we discuss an under-appreciated opportunity to extend video generation …

abstract arxiv cs.ai cs.cv data decision decision making entertainment impact internet language language models making media next prediction scale self-supervised learning supervised learning support text through token type video video data video generation world