Compressing Transformer-based self-supervised models for speech processing. (arXiv:2211.09949v1 [cs.CL])

Despite the success of Transformers in self-supervised learning with
applications to various downstream tasks, the computational cost of training
and inference remains a major challenge for applying these models to a wide
spectrum of devices. Several isolated attempts have been made to compress
Transformers, prior to applying them to downstream tasks. In this work, we aim
to provide context for the isolated results, studying several commonly used
compression techniques, including weight pruning, head pruning, low-rank
approximation, and knowledge distillation. We …

arxiv processing speech speech processing transformer