Grounding Language Models for Visual Entity Recognition

arXiv:2402.18695v1 Announce Type: new
Abstract: We introduce AutoVER, an Autoregressive model for Visual Entity Recognition. Our model extends an autoregressive Multi-modal Large Language Model by employing retrieval augmented constrained generation. It mitigates low performance on out-of-domain entities while excelling in queries that require visually-situated reasoning. Our method learns to distinguish similar entities within a vast label space by contrastively training on hard negative pairs in parallel with a sequence-to-sequence objective without an external retriever. During inference, a list of retrieved …

abstract arxiv autoregressive model cs.cl cs.cv domain language language model language models large language large language model low modal multi-modal performance queries reasoning recognition retrieval type visual