Your fairness may vary: Group fairness of pretrained language models in toxic text classification. (arXiv:2108.01250v2 [cs.CL] UPDATED)

The popularity of pretrained language models in natural language processing
systems calls for a careful evaluation of such models in down-stream tasks,
which have a higher potential for societal impact. The evaluation of such
systems usually focuses on \textit{accuracy measures}. Our findings in this
paper call for attention to be paid to \textit{fairness measures} as well.
Through the analysis of more than a dozen pretrained language models of varying
sizes on two toxic text classification tasks (English), we demonstrate that …

arxiv classification fairness language language models text text classification