Generating Illustrated Instructions

arXiv:2312.04552v2 Announce Type: replace-cross
Abstract: We introduce the new task of generating Illustrated Instructions, i.e., visual instructions customized to a user's needs. We identify desiderata unique to this task, and formalize it through a suite of automatic and human evaluation metrics, designed to measure the validity, consistency, and efficacy of the generations. We combine the power of large language models (LLMs) together with strong text-to-image generation diffusion models to propose a simple approach called StackedDiffusion, which generates such illustrated instructions …

abstract arxiv cs.ai cs.cv cs.lg cs.mm evaluation evaluation metrics human identify metrics power through type visual