Benchmarking Algorithmic Bias in Face Recognition: An Experimental Approach Using Synthetic Faces and Human Evaluation. (arXiv:2308.05441v1 [cs.CV])

We propose an experimental method for measuring bias in face recognition
systems. Existing methods to measure bias depend on benchmark datasets that are
collected in the wild and annotated for protected (e.g., race, gender) and
non-protected (e.g., pose, lighting) attributes. Such observational datasets
only permit correlational conclusions, e.g., "Algorithm A's accuracy is
different on female and male faces in dataset X.". By contrast, experimental
methods manipulate attributes individually and thus permit causal conclusions,
e.g., "Algorithm A's accuracy is affected by …

algorithmic bias arxiv benchmark benchmarking bias datasets evaluation experimental face face recognition gender human lighting race recognition synthetic systems