Evaluating Psychological Safety of Large Language Models

arXiv:2212.10529v3 Announce Type: replace
Abstract: In this work, we designed unbiased prompts to systematically evaluate the psychological safety of large language models (LLMs). First, we tested five different LLMs by using two personality tests: Short Dark Triad (SD-3) and Big Five Inventory (BFI). All models scored higher than the human average on SD-3, suggesting a relatively darker personality pattern. Despite being instruction fine-tuned with safety metrics to reduce toxicity, InstructGPT, GPT-3.5, and GPT-4 still showed dark personality patterns; these models …

abstract arxiv big cs.ai cs.cl cs.cy five human inventory language language models large language large language models llms personality prompts safety tests type unbiased work