Aligning Step-by-Step Instructional Diagrams to Video Demonstrations

arXiv:2303.13800v3 Announce Type: replace
Abstract: Multimodal alignment facilitates the retrieval of instances from one modality when queried using another. In this paper, we consider a novel setting where such an alignment is between (i) instruction steps that are depicted as assembly diagrams (commonly seen in Ikea assembly manuals) and (ii) video segments from in-the-wild videos; these videos comprising an enactment of the assembly actions in the real world. To learn this alignment, we introduce a novel supervised contrastive learning method …

abstract alignment arxiv assembly cs.cv diagrams ikea instances multimodal novel paper retrieval step-by-step type video