Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordance

arXiv:2403.18036v1 Announce Type: new
Abstract: Despite significant advancements in text-to-motion synthesis, generating language-guided human motion within 3D environments poses substantial challenges. These challenges stem primarily from (i) the absence of powerful generative models capable of jointly modeling natural language, 3D scenes, and human motion, and (ii) the generative models' intensive data requirements contrasted with the scarcity of comprehensive, high-quality, language-scene-motion datasets. To tackle these issues, we introduce a novel two-stage framework that employs scene affordance as an intermediate representation, effectively …

3d scenes abstract arxiv challenges cs.cv environments generative generative models human language modeling natural natural language stem synthesis text type