VideoPoet: A Large Language Model for Zero-Shot Video Generation

arXiv:2312.14125v2 Announce Type: replace
Abstract: We present VideoPoet, a language model capable of synthesizing high-quality video, with matching audio, from a large variety of conditioning signals. VideoPoet employs a decoder-only transformer architecture that processes multimodal inputs -- including images, videos, text, and audio. The training protocol follows that of Large Language Models (LLMs), consisting of two stages: pretraining and task-specific adaptation. During pretraining, VideoPoet incorporates a mixture of multimodal generative objectives within an autoregressive Transformer framework. The pretrained LLM serves …

arxiv cs.ai cs.cv language language model large language large language model type video video generation videopoet zero-shot