Why Not Use Your Textbook? Knowledge-Enhanced Procedure Planning of Instructional Videos

arXiv:2403.02782v1 Announce Type: new
Abstract: In this paper, we explore the capability of an agent to construct a logical sequence of action steps, thereby assembling a strategic procedural plan. This plan is crucial for navigating from an initial visual observation to a target visual outcome, as depicted in real-life instructional videos. Existing works have attained partial success by extensively leveraging various sources of information available in the datasets, such as heavy intermediate visual observations, procedural names, or natural language step-by-step …

abstract agent arxiv capability construct cs.cv explore knowledge life observation paper planning textbook type videos visual