Probing Cross-modal Semantics Alignment Capability from the Textual Perspective. (arXiv:2210.09550v1 [cs.CL])

In recent years, vision and language pre-training (VLP) models have advanced
the state-of-the-art results in a variety of cross-modal downstream tasks.
Aligning cross-modal semantics is claimed to be one of the essential
capabilities of VLP models. However, it still remains unclear about the inner
working mechanism of alignment in VLP models. In this paper, we propose a new
probing method that is based on image captioning to first empirically study the
cross-modal semantics alignment of VLP models. Our probing method …

alignment arxiv perspective semantics