Discffusion: Discriminative Diffusion Models as Few-shot Vision and Language Learners

arXiv:2305.10722v3 Announce Type: replace
Abstract: Diffusion models, such as Stable Diffusion, have shown incredible performance on text-to-image generation. Since text-to-image generation often requires models to generate visual concepts with fine-grained details and attributes specified in text prompts, can we leverage the powerful representations learned by pre-trained diffusion models for discriminative tasks such as image-text matching? To answer this question, we propose a novel approach, Discriminative Stable Diffusion (DSD), which turns pre-trained text-to-image diffusion models into few-shot discriminative learners. Our approach …

abstract arxiv concepts cs.cv diffusion diffusion models few-shot fine-grained generate image image generation language performance prompts stable diffusion text text-to-image type vision visual visual concepts