Learning to Visually Connect Actions and their Effects

arXiv:2401.10805v2 Announce Type: replace-cross
Abstract: In this work, we introduce the novel concept of visually Connecting Actions and Their Effects (CATE) in video understanding. CATE can have applications in areas like task planning and learning from demonstration. We identify and explore two different aspects of the concept of CATE: Action Selection and Effect-Affinity Assessment, where video understanding models connect actions and effects at semantic and fine-grained levels, respectively. We observe that different formulations produce representations capturing intuitive action properties. We …

abstract applications arxiv concept cs.ai cs.cv cs.lg cs.ro effects explore identify novel planning type understanding video video understanding work