Clockwork Diffusion: Efficient Generation With Model-Step Distillation

arXiv:2312.08128v2 Announce Type: replace
Abstract: This work aims to improve the efficiency of text-to-image diffusion models. While diffusion models use computationally expensive UNet-based denoising operations in every generation step, we identify that not all operations are equally relevant for the final output quality. In particular, we observe that UNet layers operating on high-res feature maps are relatively sensitive to small perturbations. In contrast, low-res feature maps influence the semantic layout of the final image and can often be perturbed with …

abstract arxiv clockwork cs.cv denoising diffusion diffusion models distillation efficiency every identify image image diffusion observe operations quality text text-to-image type unet work