LLM-grounded Video Diffusion Models

arXiv:2309.17444v3 Announce Type: replace
Abstract: Text-conditioned diffusion models have emerged as a promising tool for neural video generation. However, current models still struggle with intricate spatiotemporal prompts and often generate restricted or incorrect motion. To address these limitations, we introduce LLM-grounded Video Diffusion (LVD). Instead of directly generating videos from the text inputs, LVD first leverages a large language model (LLM) to generate dynamic scene layouts based on the text inputs and subsequently uses the generated layouts to guide a …

abstract arxiv cs.ai cs.cl cs.cv current diffusion diffusion models generate however inputs limitations llm prompts struggle text tool type video video diffusion video generation videos