AutoDistill: an End-to-End Framework to Explore and Distill Hardware-Efficient Language Models. (arXiv:2201.08539v1 [cs.LG])

Web: http://arxiv.org/abs/2201.08539

Recently, large pre-trained models have significantly improved the
performance of various Natural LanguageProcessing (NLP) tasks but they are
expensive to serve due to long serving latency and large memory usage. To
compress these models, knowledge distillation has attracted an increasing
amount of interest as one of the most effective methods for model compression.
However, existing distillation methods have not yet addressed the unique
challenges of model serving in datacenters, such as handling fast evolving
models, considering serving performance, and optimizing …

arxiv framework hardware language language models models