SSM Meets Video Diffusion Models: Efficient Video Generation with Structured State Spaces

arXiv:2403.07711v2 Announce Type: replace
Abstract: Given the remarkable achievements in image generation through diffusion models, the research community has shown increasing interest in extending these models to video generation. Recent diffusion models for video generation have predominantly utilized attention layers to extract temporal features. However, attention layers are limited by their memory consumption, which increases quadratically with the length of the sequence. This limitation presents significant challenges when attempting to generate longer video sequences using diffusion models. To overcome this …

arxiv cs.ai cs.cv diffusion diffusion models spaces ssm state type video video diffusion video generation