When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks

arXiv:2305.06626v5 Announce Type: replace
Abstract: Though majority vote among annotators is typically used for ground truth labels in natural language processing, annotator disagreement in tasks such as hate speech detection may reflect differences in opinion across groups, not noise. Thus, a crucial problem in hate speech detection is determining whether a statement is offensive to the demographic group that it targets, when that group may constitute a small fraction of the annotator pool. We construct a model that predicts individual …

abstract arxiv cs.ai cs.cl detection differences hate speech hate speech detection labels language language processing modeling natural natural language natural language processing noise opinion processing speech tasks truth type vote