Accurate inference of crowdsourcing properties when using efficient allocation strategies. (arXiv:1903.03104v2 [cs.LG] UPDATED)

Allocation strategies improve the efficiency of crowdsourcing by decreasing
the work needed to complete individual tasks accurately. However, these
algorithms introduce bias by preferentially allocating workers onto easy tasks,
leading to sets of completed tasks that are no longer representative of all
tasks. This bias challenges inference of problem-wide properties such as
typical task difficulty or crowd properties such as worker completion times,
important information that goes beyond the crowd responses themselves. Here we
study inference about problem properties when …

arxiv crowdsourcing inference strategies