EnCodecMAE: Leveraging neural codecs for universal audio representation learning

arXiv:2309.07391v2 Announce Type: replace-cross
Abstract: The goal of universal audio representation learning is to obtain foundational models that can be used for a variety of downstream tasks involving speech, music and environmental sounds. To approach this problem, methods inspired by works on self-supervised learning for NLP, like BERT, or computer vision, like masked autoencoders (MAE), are often adapted to the audio domain. In this work, we propose masking representations of the audio signal, and training a MAE to reconstruct the …

abstract arxiv audio bert computer computer vision cs.lg cs.sd eess.as environmental foundational foundational models music nlp replace representation representation learning self-supervised learning speech supervised learning tasks type universal vision