Fake Alignment: Are LLMs Really Aligned Well?

arXiv:2311.05915v3 Announce Type: replace
Abstract: The growing awareness of safety concerns in large language models (LLMs) has sparked considerable interest in the evaluation of safety. This study investigates an under-explored issue about the evaluation of LLMs, namely the substantial discrepancy in performance between multiple-choice questions and open-ended questions. Inspired by research on jailbreak attack patterns, we argue this is caused by mismatched generalization. That is, LLM only remembers the answer style for open-ended safety questions, which makes it unable to …

alignment arxiv cs.ai cs.cl fake llms type