Pushing the Limits of Zero-shot End-to-End Speech Translation

arXiv:2402.10422v1 Announce Type: new
Abstract: Data scarcity and the modality gap between the speech and text modalities are two major obstacles of end-to-end Speech Translation (ST) systems, thus hindering their performance. Prior work has attempted to mitigate these challenges by leveraging external MT data and optimizing distance metrics that bring closer the speech-text representations. However, achieving competitive results typically requires some ST data. For this reason, we introduce ZeroSwot, a method for zero-shot ST that bridges the modality gap without …

abstract arxiv challenges cs.cl data gap major metrics obstacles performance prior speech systems text translation type work zero-shot