Multilingual CheckList: Generation and Evaluation. (arXiv:2203.12865v3 [cs.CL] UPDATED)

Multilingual evaluation benchmarks usually contain limited high-resource
languages and do not test models for specific linguistic capabilities.
CheckList is a template-based evaluation approach that tests models for
specific capabilities. The CheckList template creation process requires native
speakers, posing a challenge in scaling to hundreds of languages. In this work,
we explore multiple approaches to generate Multilingual CheckLists. We device
an algorithm - Template Extraction Algorithm (TEA) for automatically extracting
target language CheckList templates from machine translated instances of a
source …

arxiv checklist evaluation