Beyond prompt brittleness: Evaluating the reliability and consistency of political worldviews in LLMs

arXiv:2402.17649v1 Announce Type: new
Abstract: Due to the widespread use of large language models (LLMs) in ubiquitous systems, we need to understand whether they embed a specific worldview and what these views reflect. Recent studies report that, prompted with political questionnaires, LLMs show left-liberal leanings. However, it is as yet unclear whether these leanings are reliable (robust to prompt variations) and whether the leaning is consistent across policies and political leaning. We propose a series of tests which assess the …

abstract arxiv beyond cs.cl cs.cy embed language language models large language large language models llms political prompt reliability report show studies systems type