Can depth-adaptive BERT perform better on binary classification tasks. (arXiv:2111.10951v2 [cs.CL] UPDATED)

In light of the success of transferring language models into NLP tasks, we
ask whether the full BERT model is always the best and does it exist a simple
but effective method to find the winning ticket in state-of-the-art deep neural
networks without complex calculations. We construct a series of BERT-based
models with different size and compare their predictions on 8 binary
classification tasks. The results show there truly exist smaller sub-networks
performing better than the full model. Then we …

arxiv bert classification