ILLUME: Rationalizing Vision-Language Models by Interacting with their Jabber. (arXiv:2208.08241v2 [cs.LG] UPDATED)

Bootstrapping from pre-trained language models has been proven to be an
efficient approach for building foundation vision-language models (VLM) for
tasks such as image captioning or visual question answering. However, it is
difficult-if not impossible-to utilize it to make the model conform with user's
rationales for specific answers. To elicit and reinforce commonsense reasons,
we propose an iterative sampling and tuning paradigm, called ILLUME, that
executes the following loop: Given an image-question-answer prompt, the VLM
samples multiple candidate rationales, and …

arxiv language language models lg vision