Measuring Social Biases in Masked Language Models by Proxy of Prediction Quality

arXiv:2402.13954v1 Announce Type: new
Abstract: Social and political scientists often aim to discover and measure distinct biases from text data representations (embeddings). Innovative transformer-based language models produce contextually-aware token embeddings and have achieved state-of-the-art performance for a variety of natural language tasks, but have been shown to encode unwanted biases for downstream applications. In this paper, we evaluate the social biases encoded by transformers trained with the masked language modeling objective using proposed proxy functions within an iterative masking experiment …

abstract aim art arxiv biases cs.cl data embeddings encode language language models measuring natural natural language performance political prediction quality scientists social state tasks text token transformer type