Diffusion Models Trained with Large Data Are Transferable Visual Models

arXiv:2403.06090v1 Announce Type: new
Abstract: We show that, simply initializing image understanding models using a pre-trained UNet (or transformer) of diffusion models, it is possible to achieve remarkable transferable performance on fundamental vision perception tasks using a moderate amount of target data (even synthetic data only), including monocular depth, surface normal, image segmentation, matting, human pose estimation, among virtually many others. Previous works have adapted diffusion models for various perception tasks, often reformulating these tasks as generation processes to align …

abstract arxiv cs.cv data diffusion diffusion models image perception performance show surface synthetic synthetic data tasks transformer type understanding unet vision visual