Corpus Considerations for Annotator Modeling and Scaling

arXiv:2404.02340v1 Announce Type: new
Abstract: Recent trends in natural language processing research and annotation tasks affirm a paradigm shift from the traditional reliance on a single ground truth to a focus on individual perspectives, particularly in subjective tasks. In scenarios where annotation tasks are meant to encompass diversity, models that solely rely on the majority class labels may inadvertently disregard valuable minority perspectives. This oversight could result in the omission of crucial information and, in a broader context, risk disrupting …

abstract affirm annotation arxiv cs.cl diversity focus language language processing modeling natural natural language natural language processing paradigm perspectives processing reliance research scaling shift tasks trends truth type