The Rediscovery Hypothesis: Language Models Need to Meet Linguistics. (arXiv:2103.01819v2 [cs.CL] UPDATED)

There is an ongoing debate in the NLP community whether modern language
models contain linguistic knowledge, recovered through so-called probes. In
this paper, we study whether linguistic knowledge is a necessary condition for
the good performance of modern language models, which we call the
\textit{rediscovery hypothesis}. In the first place, we show that language
models that are significantly compressed but perform well on their pretraining
objectives retain good scores when probed for linguistic structures. This
result supports the rediscovery hypothesis …

arxiv hypothesis language language models linguistics