Prejudice and Caprice: A Statistical Framework for Measuring Social Discrimination in Large Language Models

arXiv:2402.15481v1 Announce Type: new
Abstract: The growing integration of large language models (LLMs) into social operations amplifies their impact on decisions in crucial areas such as economics, law, education, and healthcare, raising public concerns about these models' discrimination-related safety and reliability. However, prior discrimination measuring frameworks solely assess the average discriminatory behavior of LLMs, often proving inadequate due to the overlook of an additional discrimination-leading factor, i.e., the LLMs' prediction variation across diverse contexts. In this work, we present the …

abstract arxiv concerns cs.cl cs.cy decisions discrimination economics education framework frameworks healthcare impact integration language language models large language large language models law llms measuring operations prior public reliability safety social statistical type