Stick to your Role! Stability of Personal Values Expressed in Large Language Models

arXiv:2402.14846v1 Announce Type: cross
Abstract: The standard way to study Large Language Models (LLMs) through benchmarks or psychology questionnaires is to provide many different queries from similar minimal contexts (e.g. multiple choice questions). However, due to LLM's highly context-dependent nature, conclusions from such minimal-context evaluations may be little informative about the model's behavior in deployment (where it will be exposed to many new contexts). We argue that context-dependence should be studied as another dimension of LLM comparison alongside others such …

abstract arxiv benchmarks context cs.ai cs.cl cs.lg language language models large language large language models llm llms multiple nature psychology queries questions role stability standard study through type values