AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agents

arXiv:2403.12835v1 Announce Type: new
Abstract: Traditional approaches in physics-based motion generation, centered around imitation learning and reward shaping, often struggle to adapt to new scenarios. To tackle this limitation, we propose AnySkill, a novel hierarchical method that learns physically plausible interactions following open-vocabulary instructions. Our approach begins by developing a set of atomic actions via a low-level controller trained via imitation learning. Upon receiving an open-vocabulary textual instruction, AnySkill employs a high-level policy that selects and integrates these atomic actions …

abstract adapt agents arxiv cs.cv cs.ro hierarchical imitation learning interactions interactive novel physics set struggle type