Towards Measuring the Representation of Subjective Global Opinions in Language Models

arXiv:2306.16388v2 Announce Type: replace
Abstract: Large language models (LLMs) may not equitably represent diverse global perspectives on societal issues. In this paper, we develop a quantitative framework to evaluate whose opinions model-generated responses are more similar to. We first build a dataset, GlobalOpinionQA, comprised of questions and answers from cross-national surveys designed to capture diverse opinions on global issues across different countries. Next, we define a metric that quantifies the similarity between LLM-generated survey responses and human responses, conditioned on …

arxiv cs.ai cs.cl global language language models measuring opinions representation type