TIPAA-SSL: Text Independent Phone-to-Audio Alignment based on Self-Supervised Learning and Knowledge Transfer

arXiv:2405.02124v1 Announce Type: cross
Abstract: In this paper, we present a novel approach for text independent phone-to-audio alignment based on phoneme recognition, representation learning and knowledge transfer. Our method leverages a self-supervised model (wav2vec2) fine-tuned for phoneme recognition using a Connectionist Temporal Classification (CTC) loss, a dimension reduction model and a frame-level phoneme classifier trained thanks to forced-alignment labels (using Montreal Forced Aligner) to produce multi-lingual phonetic representations, thus requiring minimal additional training. We evaluate our model using synthetic native …

abstract alignment arxiv audio classification cs.ai cs.cl cs.lg eess.as independent knowledge loss novel paper phone recognition representation representation learning self-supervised learning ssl supervised learning temporal text transfer type wav2vec2