Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models

arXiv:2305.10474v3 Announce Type: replace-cross
Abstract: Despite tremendous progress in generating high-quality images using diffusion models, synthesizing a sequence of animated frames that are both photorealistic and temporally coherent is still in its infancy. While off-the-shelf billion-scale datasets for image generation are available, collecting similar video data of the same scale is still challenging. Also, training a video diffusion model is computationally much more expensive than its image counterpart. In this work, we explore finetuning a pretrained image diffusion model with …

abstract animated arxiv billion correlation cs.cv cs.gr cs.lg data datasets diffusion diffusion models image image generation images noise photorealistic prior progress quality scale type video video data video diffusion