Choose Your Lenses: Flaws in Gender Bias Evaluation. (arXiv:2210.11471v1 [cs.CL])

Considerable efforts to measure and mitigate gender bias in recent years have
led to the introduction of an abundance of tasks, datasets, and metrics used in
this vein. In this position paper, we assess the current paradigm of gender
bias evaluation and identify several flaws in it. First, we highlight the
importance of extrinsic bias metrics that measure how a model's performance on
some task is affected by gender, as opposed to intrinsic evaluations of model
representations, which are less …

arxiv bias evaluation flaws gender gender bias