Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models

arXiv:2312.03052v2 Announce Type: replace
Abstract: Solving complex visual tasks such as "Who invented the musical instrument on the right?" involves a composition of skills: understanding space, recognizing instruments, and also retrieving prior knowledge. Recent work shows promise by decomposing such tasks using a large language model (LLM) into an executable program that invokes specialized vision models. However, generated programs are error-prone: they omit necessary steps, include spurious ones, and are unable to recover when the specialized models give incorrect outputs. …

abstract arxiv cs.cl cs.cv distillation knowledge language language model language models large language large language model llm prior programmatic reasoning shows skills space tasks tools type understanding vision vision-language models visual work