LMentry: A Language Model Benchmark of Elementary Language Tasks. (arXiv:2211.02069v1 [cs.CL])

As the performance of large language models rapidly improves, benchmarks are
getting larger and more complex as well. We present LMentry, a benchmark that
avoids this "arms race" by focusing on a compact set of tasks that are trivial
to humans, e.g. writing a sentence containing a specific word, identifying
which words in a list belong to a specific category, or choosing which of two
words is longer. LMentry is specifically designed to provide quick and
interpretable insights into the …

arxiv benchmark elementary language language model