Crowdsourcing Relative Rankings of Multi-Word Expressions: Experts versus Non-Experts. (arXiv:2206.08724v1 [cs.CL])

In this study we investigate to which degree experts and non-experts agree on
questions of difficulty in a crowdsourcing experiment. We ask non-experts
(second language learners of Swedish) and two groups of experts (teachers of
Swedish as a second/foreign language and CEFR experts) to rank multi-word
expressions in a crowdsourcing experiment. We find that the resulting rankings
by all the three tested groups correlate to a very high degree, which suggests
that judgments produced in a comparative setting are not …

arxiv crowdsourcing experts