A structured regression approach for evaluating model performance across intersectional subgroups

arXiv:2401.14893v2 Announce Type: replace
Abstract: Disaggregated evaluation is a central task in AI fairness assessment, where the goal is to measure an AI system's performance across different subgroups defined by combinations of demographic or other sensitive attributes. The standard approach is to stratify the evaluation data across subgroups and compute performance metrics separately for each group. However, even for moderately-sized evaluation datasets, sample sizes quickly get small once considering intersectional subgroups, which greatly limits the extent to which intersectional groups …

abstract ai system arxiv assessment attributes cs.cy cs.lg data evaluation fairness performance regression replace s performance standard stat.ap stat.ml subgroups type