Open Implementation and Study of BEST-RQ for Speech Processing

arXiv:2405.04296v1 Announce Type: cross
Abstract: Self-Supervised Learning (SSL) has proven to be useful in various speech tasks. However, these methods are generally very demanding in terms of data, memory, and computational resources. BERT-based Speech pre-Training with Random-projection Quantizer (BEST-RQ), is an SSL method that has shown great performance on Automatic Speech Recognition (ASR) while being simpler than other SSL methods, such as wav2vec 2.0. Despite BEST-RQ's great performance, details are lacking in the original paper, such as the amount of …

abstract arxiv bert computational cs.cl cs.lg data however implementation memory performance pre-training processing projection random resources self-supervised learning speech speech processing ssl study supervised learning tasks terms training type