Composing Object Relations and Attributes for Image-Text Matching

arXiv:2406.11820v1 Announce Type: new
Abstract: We study the visual semantic embedding problem for image-text matching. Most existing work utilizes a tailored cross-attention mechanism to perform local alignment across the two image and text modalities. This is computationally expensive, even though it is more powerful than the unimodal dual-encoder approach. This work introduces a dual-encoder image-text matching model, leveraging a scene graph to represent captions with nodes for objects and attributes interconnected by relational edges. Utilizing a graph attention network, our …

abstract alignment arxiv attention attributes cs.cv embedding encoder image object problem relations semantic study text type visual work