TagAlign: Improving Vision-Language Alignment with Multi-Tag Classification

arXiv:2312.14149v3 Announce Type: replace
Abstract: The crux of learning vision-language models is to extract semantically aligned information from visual and linguistic data. Existing attempts usually face the problem of coarse alignment, e.g., the vision encoder struggles in localizing an attribute-specified object. In this work, we propose an embarrassingly simple approach to better align image and text features with no need of additional data formats other than image-text pairs. Concretely, given an image and its paired text, we manage to parse …

alignment arxiv classification cs.ai cs.cv improving language tag type vision