ArcSin: Adaptive ranged cosine Similarity injected noise for Language-Driven Visual Tasks

arXiv:2402.17298v1 Announce Type: new
Abstract: In this study, we address the challenging task of bridging the modality gap between learning from language and inference for visual tasks, including Visual Question Answering (VQA), Image Captioning (IC) and Visual Entailment (VE). We train models for these tasks in a zero-shot cross-modal transfer setting, a domain where the previous state-of-the-art method relied on the fixed scale noise injection, often compromising the semantic content of the original modality embedding. To combat it, we propose …

abstract arxiv captioning cs.cv gap image inference language modal noise question question answering study tasks train type visual vqa zero-shot