Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective

arXiv:2406.14023v1 Announce Type: new
Abstract: As Large Language Models (LLMs) become an important way of information seeking, there have been increasing concerns about the unethical content LLMs may generate. In this paper, we conduct a rigorous evaluation of LLMs' implicit bias towards certain groups by attacking them with carefully crafted instructions to elicit biased responses. Our attack methodology is inspired by psychometric principles in cognitive and social psychology. We propose three attack approaches, i.e., Disguise, Deception, and Teaching, based on …

abstract arxiv become bias concerns cs.ai cs.cl evaluation generate important information language language models large language large language models llms paper perspective them type