Non-autoregressive Sequence-to-Sequence Vision-Language Models

arXiv:2403.02249v1 Announce Type: new
Abstract: Sequence-to-sequence vision-language models are showing promise, but their applicability is limited by their inference latency due to their autoregressive way of generating predictions. We propose a parallel decoding sequence-to-sequence vision-language model, trained with a Query-CTC loss, that marginalizes over multiple inference paths in the decoder. This allows us to model the joint distribution of tokens, rather than restricting to conditional distribution as in an autoregressive model. The resulting model, NARVL, achieves performance on-par with its …

abstract arxiv cs.ai cs.cv decoder decoding inference inference latency language language model language models latency loss multiple predictions query the decoder type vision vision-language models