WorldGPT: A Sora-Inspired Video AI Agent as Rich World Models from Text and Image Inputs

arXiv:2403.07944v1 Announce Type: new
Abstract: Several text-to-video diffusion models have demonstrated commendable capabilities in synthesizing high-quality video content. However, it remains a formidable challenge pertaining to maintaining temporal consistency and ensuring action smoothness throughout the generated sequences. In this paper, we present an innovative video generation AI agent that harnesses the power of Sora-inspired multimodal learning to build skilled world models framework based on textual prompts and accompanying images. The framework includes two parts: prompt enhancer and full video translation. …

abstract agent arxiv capabilities challenge cs.ai cs.cv diffusion diffusion models generated however image inputs paper quality sora temporal text text-to-video type video video ai video diffusion world world models