Exploring Extreme Parameter Compression for Pre-trained Language Models. (arXiv:2205.10036v1 [cs.CL])

Recent work explored the potential of large-scale Transformer-based
pre-trained models, especially Pre-trained Language Models (PLMs) in natural
language processing. This raises many concerns from various perspectives, e.g.,
financial costs and carbon emissions. Compressing PLMs like BERT with
negligible performance loss for faster inference and cheaper deployment has
attracted much attention. In this work, we aim to explore larger compression
ratios for PLMs, among which tensor decomposition is a potential but
under-investigated one. Two decomposition and reconstruction protocols are
further proposed …

arxiv compression language language models