FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models

arXiv:2403.20105v1 Announce Type: new
Abstract: Foundation models have exhibited unprecedented capabilities in tackling many domains and tasks. Models such as CLIP are currently widely used to bridge cross-modal representations, and text-to-image diffusion models are arguably the leading models in terms of realistic image generation. Image generative models are trained on massive datasets that provide them with powerful internal spatial representations. In this work, we explore the potential benefits of such representations, beyond image generation, in particular, for dense visual prediction …

arxiv cs.cv diff diffusion diffusion models free segmentation training type