DirecT2V: Large Language Models are Frame-Level Directors for Zero-Shot Text-to-Video Generation

In the paradigm of AI-generated content (AIGC), there has been increasing attention to transferring knowledge from pre-trained text-to-image (T2I) models to text-to-video (T2V) generation. Despite their effectiveness, these frameworks face challenges in maintaining consistent narratives and handling shifts in scene composition or object placement from a single abstract user prompt. Exploring the ability of large language models (LLMs) to generate time-dependent, frame-by-frame prompts, this paper introduces a new framework, dubbed DirecT2V. DirecT2V leverages instruction-tuned LLMs as directors, enabling the inclusion …

abstract aigc ai-generated content attention challenges consistent cs.ai cs.cl cs.cv face frameworks generated image knowledge language language models large language large language models paradigm placement text text-to-image text-to-video video video generation zero-shot