How many labelers do you have? A closer look at gold-standard labels

arXiv:2206.12041v2 Announce Type: replace-cross
Abstract: The construction of most supervised learning datasets revolves around collecting multiple labels for each instance, then aggregating the labels to form a type of "gold-standard". We question the wisdom of this pipeline by developing a (stylized) theoretical model of this process and analyzing its statistical consequences, showing how access to non-aggregated label information can make training well-calibrated models more feasible than it is with gold-standard labels. The entire story, however, is subtle, and the contrasts …

abstract arxiv closer look construction cs.hc cs.lg datasets form gold instance labels look math.st multiple pipeline process question replace standard stat.th supervised learning type you