Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation

arXiv:2404.06542v1 Announce Type: new
Abstract: Open-vocabulary semantic segmentation aims at segmenting arbitrary categories expressed in textual form. Previous works have trained over large amounts of image-caption pairs to enforce pixel-level multimodal alignments. However, captions provide global information about the semantics of a given image but lack direct localization of individual concepts. Further, training on large-scale datasets inevitably brings significant computational costs. In this paper, we propose FreeDA, a training-free diffusion-augmented method for open-vocabulary semantic segmentation, which leverages the ability of …

abstract arxiv captions concepts cs.cv diffusion form free global however image information localization multimodal offline pixel segmentation semantic semantics textual training type