The CLIP Model is Secretly an Image-to-Prompt Converter

arXiv:2305.12716v2 Announce Type: replace
Abstract: The Stable Diffusion model is a prominent text-to-image generation model that relies on a text prompt as its input, which is encoded using the Contrastive Language-Image Pre-Training (CLIP). However, text prompts have limitations when it comes to incorporating implicit information from reference images. Existing methods have attempted to address this limitation by employing expensive training procedures involving millions of training samples for image-to-image generation. In contrast, this paper demonstrates that the CLIP model, as utilized …

abstract arxiv clip converter cs.cv diffusion diffusion model image image generation images information language limitations pre-training prompt prompts reference stable diffusion text text-to-image training type