Spot Check Equivalence: an Interpretable Metric for Information Elicitation Mechanisms

arXiv:2402.13567v1 Announce Type: new
Abstract: Because high-quality data is like oxygen for AI systems, effectively eliciting information from crowdsourcing workers has become a first-order problem for developing high-performance machine learning algorithms. Two prevalent paradigms, spot-checking and peer prediction, enable the design of mechanisms to evaluate and incentivize high-quality data from human labelers. So far, at least three metrics have been proposed to compare the performances of these techniques [33, 8, 3]. However, different metrics lead to divergent and even contradictory …

abstract ai systems algorithms arxiv become check crowdsourcing cs.ai cs.gt cs.lg data design information machine machine learning machine learning algorithms peer performance prediction quality quality data spot systems type workers