Getting More for Less: Using Weak Labels and AV-Mixup for Robust Audio-Visual Speaker Verification

arXiv:2309.07115v2 Announce Type: replace-cross
Abstract: Distance Metric Learning (DML) has typically dominated the audio-visual speaker verification problem space, owing to strong performance in new and unseen classes. In our work, we explored multitask learning techniques to further enhance DML, and show that an auxiliary task with even weak labels can increase the quality of the learned speaker representation without increasing model complexity during inference. We also extend the Generalized End-to-End Loss (GE2E) to multimodal inputs and demonstrate that it can …

abstract arxiv audio cs.cv cs.lg cs.mm cs.sd dml eess.as labels multitask learning performance problem replace robust show space speaker type verification visual work