ReGround: Improving Textual and Spatial Grounding at No Cost

arXiv:2403.13589v1 Announce Type: new
Abstract: When an image generation process is guided by both a text prompt and spatial cues, such as a set of bounding boxes, do these elements work in harmony, or does one dominate the other? Our analysis of a pretrained image diffusion model that integrates gated self-attention into the U-Net reveals that spatial grounding often outweighs textual grounding due to the sequential flow from gated self-attention to cross-attention. We demonstrate that such bias can be significantly …

abstract analysis arxiv attention cost cs.cv diffusion diffusion model image image diffusion image generation image generation process improving process prompt self-attention set spatial text textual type work