Transcription and translation of videos using fine-tuned XLSR Wav2Vec2 on custom dataset and mBART

arXiv:2403.00212v1 Announce Type: cross
Abstract: This research addresses the challenge of training an ASR model for personalized voices with minimal data. Utilizing just 14 minutes of custom audio from a YouTube video, we employ Retrieval-Based Voice Conversion (RVC) to create a custom Common Voice 16.0 corpus. Subsequently, a Cross-lingual Self-supervised Representations (XLSR) Wav2Vec2 model is fine-tuned on this dataset. The developed web-based GUI efficiently transcribes and translates input Hindi videos. By integrating XLSR Wav2Vec2 and mBART, the system aligns the …

abstract arxiv asr audio challenge conversion cs.cl cs.cv cs.lg cs.sd data dataset eess.as personalized research retrieval training transcription translation type video videos voice voices wav2vec2 youtube