HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Grounding

arXiv:2404.13400v1 Announce Type: new
Abstract: Visual grounding, which aims to ground a visual region via natural language, is a task that heavily relies on cross-modal alignment. Existing works utilized uni-modal pre-trained models to transfer visual/linguistic knowledge separately while ignoring the multimodal corresponding information. Motivated by recent advancements in contrastive language-image pre-training and low-rank adaptation (LoRA) methods, we aim to solve the grounding task based on multimodal pre-training. However, there exists significant task gaps between pre-training and grounding. Therefore, to address …

arxiv cs.cv fine-grained hierarchical multimodal type visual