Questioning the Survey Responses of Large Language Models

arXiv:2306.07951v3 Announce Type: replace
Abstract: As large language models increase in capability, researchers have started to conduct surveys of all kinds on these models in order to investigate the population represented by their responses. In this work, we critically examine language models' survey responses on the basis of the well-established American Community Survey by the U.S. Census Bureau and investigate whether they elicit a faithful representations of any human population. Using a de-facto standard multiple-choice prompting technique and evaluating 39 …

abstract arxiv capability community cs.cl language language models large language large language models population researchers responses survey surveys type work