Compact Speech Translation Models via Discrete Speech Units Pretraining

arXiv:2402.19333v1 Announce Type: new
Abstract: Using Self-Supervised Learning (SSL) as model initialization is now common to obtain strong results in Speech Translation (ST). However, they also impose a large memory footprint, hindering on-device deployment. In this paper, we leverage the SSL models by pretraining smaller models on their Discrete Speech Units (DSU). We pretrain encoder-decoder models on 1) Filterbank-to-DSU and 2) DSU-to-Translation data, and take the encoder from 1) and the decoder from 2) to initialise a new model, finetuning …

abstract arxiv cs.cl cs.sd deployment eess.as memory paper pretraining results self-supervised learning speech ssl supervised learning translation type units via