LEGO: Learning EGOcentric Action Frame Generation via Visual Instruction Tuning

arXiv:2312.03849v2 Announce Type: replace
Abstract: Generating instructional images of human daily actions from an egocentric viewpoint serves as a key step towards efficient skill transfer. In this paper, we introduce a novel problem -- egocentric action frame generation. The goal is to synthesize an image depicting an action in the user's context (i.e., action frame) by conditioning on a user prompt and an input egocentric image. Notably, existing egocentric action datasets lack the detailed annotations that describe the execution of …

arxiv cs.cv lego type via visual