Language Models Can See: Plugging Visual Controls in Text Generation. (arXiv:2205.02655v1 [cs.CV])

Web: http://arxiv.org/abs/2205.02655

Generative language models (LMs) such as GPT-2/3 can be prompted to generate
text with remarkable quality. While they are designed for text-prompted
generation, it remains an open question how the generation process could be
guided by modalities beyond text such as images. In this work, we propose a
training-free framework, called MAGIC (iMAge-Guided text generatIon with CLIP),
for plugging in visual controls in the generation process and enabling LMs to
perform multimodal tasks (e.g., image captioning) in a zero-shot manner. …

arxiv cv language language models models text text generation