Diffusion Models Generate Images Like Painters: an Analytical Theory of Outline First, Details Later

arXiv:2303.02490v2 Announce Type: replace
Abstract: How do diffusion generative models convert pure noise into meaningful images? In a variety of pretrained diffusion models (including conditional latent space models like Stable Diffusion), we observe that the reverse diffusion process that underlies image generation has the following properties: (i) individual trajectories tend to be low-dimensional and resemble 2D `rotations'; (ii) high-variance scene features like layout tend to emerge earlier, while low-variance details tend to emerge later; and (iii) early perturbations tend to …

abstract arxiv cs.ai cs.cv cs.gr cs.ne diffusion diffusion models generate generative generative models image image generation images noise observe process space stable diffusion theory type