Establishing degrees of closeness between audio recordings along different dimensions using large-scale cross-lingual models

In the highly constrained context of low-resource language studies, we explore vector representations of speech from a pretrained model to determine their level of abstraction with regard to the audio signal. We propose a new unsupervised method using ABX tests on audio recordings with carefully curated metadata to shed light on the type of information present in the representations. ABX tests determine whether the representations computed by a multilingual speech model encode a given characteristic. Three experiments are devised: one …

abstraction audio audio recordings context cross-lingual cs.cl cs.sd dimensions eess.as explore language low regard scale signal speech studies tests unsupervised vector