Understanding Multimodal Procedural Knowledge by Sequencing Multimodal Instructional Manuals

arXiv:2110.08486v4 Announce Type: replace-cross
Abstract: The ability to sequence unordered events is an essential skill to comprehend and reason about real world task procedures, which often requires thorough understanding of temporal common sense and multimodal information, as these procedures are often communicated through a combination of texts and images. Such capability is essential for applications such as sequential task planning and multi-source instruction summarization. While humans are capable of reasoning about and sequencing unordered multimodal procedural instructions, whether current machine …

abstract arxiv capability combination common sense cs.cl cs.cv events images information knowledge multimodal reason sense sequencing temporal through type understanding world