StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation

arXiv:2405.01434v1 Announce Type: new
Abstract: For recent diffusion-based generative models, maintaining consistent content across a series of generated images, especially those containing subjects and complex details, presents a significant challenge. In this paper, we propose a new way of self-attention calculation, termed Consistent Self-Attention, that significantly boosts the consistency between the generated images and augments prevalent pretrained diffusion-based text-to-image models in a zero-shot manner. To extend our method to long-range video generation, we further introduce a novel semantic space temporal …

arxiv attention consistent cs.cv image self-attention type video video generation