Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone. (arXiv:2206.07643v1 [cs.CV])

Web: http://arxiv.org/abs/2206.07643

Vision-language (VL) pre-training has recently received considerable
attention. However, most existing end-to-end pre-training approaches either
only aim to tackle VL tasks such as image-text retrieval, visual question
answering (VQA) and image captioning that test high-level understanding of
images, or only target region-level understanding for tasks such as phrase
grounding and object detection. We present FIBER (Fusion-In-the-Backbone-based
transformER), a new VL model architecture that can seamlessly handle both these
types of tasks. Instead of having dedicated transformer layers for fusion after …

arxiv cv fine fusion language pre-training training vision