Learning How To Ask: Cycle-Consistency Refines Prompts in Multimodal Foundation Models

arXiv:2402.08756v1 Announce Type: new
Abstract: When LLMs perform zero-shot inference, they typically use a prompt with a task specification, and generate a completion. However, there is no work to explore the possibility of the reverse - going from completion to task specification. In this paper, we employ both directions to perform cycle-supervised learning entirely in-context. Our goal is to create a forward map f : X -> Y (e.g. image -> generated caption), coupled with a backward map g : …

abstract arxiv cs.cl cs.cv explore foundation generate inference llms multimodal paper possibility prompt prompts type work zero-shot