ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis

arXiv:2403.17936v1 Announce Type: new
Abstract: Gestures play a key role in human communication. Recent methods for co-speech gesture generation, while managing to generate beat-aligned motions, struggle generating gestures that are semantically aligned with the utterance. Compared to beat gestures that align naturally to the audio signal, semantically coherent gestures require modeling the complex interactions between the language and human motion, and can be controlled by focusing on certain words. Therefore, we present ConvoFusion, a diffusion-based approach for multi-modal gesture synthesis, …

abstract arxiv audio communication conversational cs.cv diffusion generate gestures human key modal modeling multi-modal role signal speech struggle synthesis type