A Dataset for the Validation of Truth Inference Algorithms Suitable for Online Deployment

arXiv:2403.08826v1 Announce Type: cross
Abstract: For the purpose of efficient and cost-effective large-scale data labeling, crowdsourcing is increasingly being utilized. To guarantee the quality of data labeling, multiple annotations need to be collected for each data sample, and truth inference algorithms have been developed to accurately infer the true labels. Despite previous studies having released public datasets to evaluate the efficacy of truth inference algorithms, these have typically focused on a single type of crowdsourcing task and neglected the temporal …

abstract algorithms annotations arxiv cost crowdsourcing cs.hc cs.lg data data labeling dataset deployment inference labeling multiple quality sample scale truth type validation