Multilingual Turn-taking Prediction Using Voice Activity Projection

arXiv:2403.06487v1 Announce Type: new
Abstract: This paper investigates the application of voice activity projection (VAP), a predictive turn-taking model for spoken dialogue, on multilingual data, encompassing English, Mandarin, and Japanese. The VAP model continuously predicts the upcoming voice activities of participants in dyadic dialogue, leveraging a cross-attention Transformer to capture the dynamic interplay between participants. The results show that a monolingual VAP model trained on one language does not make good predictions when applied to other languages. However, a multilingual …

abstract application arxiv attention cs.cl cs.sd data dialogue dynamic eess.as english japanese multilingual paper prediction predictive projection spoken transformer type voice