You don't need a personality test to know these models are unreliable: Assessing the Reliability of Large Language Models on Psychometric Instruments

arXiv:2311.09718v2 Announce Type: replace
Abstract: The versatility of Large Language Models (LLMs) on natural language understanding tasks has made them popular for research in social sciences. To properly understand the properties and innate personas of LLMs, researchers have performed studies that involve using prompts in the form of questions that ask LLMs about particular opinions. In this study, we take a cautionary step back and examine whether the current format of prompting LLMs elicits responses in a consistent and robust …

abstract arxiv cs.ai cs.cl language language models language understanding large language large language models llms natural natural language personality personas popular reliability research researchers social social sciences tasks test them type understanding