Speech Emotion Recognition with Distilled Prosodic and Linguistic Affect Representations

arXiv:2309.04849v2 Announce Type: replace-cross
Abstract: We propose EmoDistill, a novel speech emotion recognition (SER) framework that leverages cross-modal knowledge distillation during training to learn strong linguistic and prosodic representations of emotion from speech. During inference, our method only uses a stream of speech signals to perform unimodal SER thus reducing computation overhead and avoiding run-time transcription and prosodic feature extraction errors. During training, our method distills information at both embedding and logit levels from a pair of pre-trained Prosodic and …

abstract arxiv cs.ai cs.cl cs.lg distillation emotion framework inference knowledge learn modal novel recognition speech speech emotion training type