Exploiting Diffusion Prior for Generalizable Dense Prediction

arXiv:2311.18832v2 Announce Type: replace
Abstract: Contents generated by recent advanced Text-to-Image (T2I) diffusion models are sometimes too imaginative for existing off-the-shelf dense predictors to estimate due to the immitigable domain gap. We introduce DMP, a pipeline utilizing pre-trained T2I models as a prior for dense prediction tasks. To address the misalignment between deterministic prediction tasks and stochastic T2I models, we reformulate the diffusion process through a sequence of interpolations, establishing a deterministic mapping between input RGB images and output prediction …

abstract advanced arxiv contents cs.cv diffusion diffusion models domain gap generated image pipeline prediction prior tasks text text-to-image type