ElitePLM: An Empirical Study on General Language Ability Evaluation of Pretrained Language Models. (arXiv:2205.01523v1 [cs.CL])

Nowadays, pretrained language models (PLMs) have dominated the majority of
NLP tasks. While, little research has been conducted on systematically
evaluating the language abilities of PLMs. In this paper, we present a
large-scale empirical study on general language ability evaluation of PLMs
(ElitePLM). In our study, we design four evaluation dimensions, i.e. memory,
comprehension, reasoning, and composition, to measure ten widely-used PLMs
within five categories. Our empirical results demonstrate that: (1) PLMs with
varying training objectives and strategies are good …

arxiv evaluation general language language models study