Step Differences in Instructional Video

arXiv:2404.16222v1 Announce Type: new
Abstract: Comparing a user video to a reference how-to video is a key requirement for AR/VR technology delivering personalized assistance tailored to the user's progress. However, current approaches for language-based assistance can only answer questions about a single video. We propose an approach that first automatically generates large amounts of visual instruction tuning data involving pairs of videos from HowTo100M by leveraging existing step annotations and accompanying narrations, and then trains a video-conditioned language model to …

abstract arxiv cs.cv current differences however how-to key language personalized progress questions reference technology type video visual