Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models

arXiv:2403.14291v1 Announce Type: new
Abstract: Diffusion models represent a new paradigm in text-to-image generation. Beyond generating high-quality images from text prompts, models such as Stable Diffusion have been successfully extended to the joint generation of semantic segmentation pseudo-masks. However, current extensions primarily rely on extracting attentions linked to prompt words used for image synthesis. This approach limits the generation of segmentation masks derived from word tokens not contained in the text prompt. In this work, we introduce Open-Vocabulary Attention Maps …

abstract arxiv attention beyond cs.cv current diffusion diffusion models extensions however image image generation images maps masks new paradigm optimization paradigm prompts quality segmentation semantic stable diffusion text text-to-image token type