Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers

arXiv:2403.07214v2 Announce Type: replace
Abstract: This paper, for the first time, explores text-to-image diffusion models for Zero-Shot Sketch-based Image Retrieval (ZS-SBIR). We highlight a pivotal discovery: the capacity of text-to-image diffusion models to seamlessly bridge the gap between sketches and photos. This proficiency is underpinned by their robust cross-modal capabilities and shape bias, findings that are substantiated through our pilot studies. In order to harness pre-trained diffusion models effectively, we introduce a straightforward yet powerful strategy focused on two key …

abstract arxiv bias bridge capabilities capacity cs.cv diffusion diffusion models discovery gap highlight image image diffusion modal paper photo photos pivotal retrieval robust sbir sketches text text-to-image type zero-shot