SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos

arXiv:2403.01599v1 Announce Type: cross
Abstract: We study the problem of procedure planning in instructional videos, which aims to make a goal-oriented sequence of action steps given partial visual state observations. The motivation of this problem is to learn a structured and plannable state and action space. Recent works succeeded in sequence modeling of steps with only sequence-level annotations accessible during training, which overlooked the roles of states in the procedures. In this work, we point out that State CHangEs MAtter …

abstract arxiv cs.ai cs.cl cs.cv cs.lg learn matter motivation planning schema space state study type videos visual