Enhancing Role-playing Systems through Aggressive Queries: Evaluation and Improvement

arXiv:2402.10618v1 Announce Type: new
Abstract: The advent of Large Language Models (LLMs) has propelled dialogue generation into new realms, particularly in the field of role-playing systems (RPSs). While enhanced with ordinary role-relevant training dialogues, existing LLM-based RPSs still struggle to align with roles when handling intricate and trapped queries in boundary scenarios. In this paper, we design the Modular ORchestrated Trap-setting Interaction SystEm (MORTISE) to benchmark and improve the role-playing LLMs' performance. MORTISE can produce highly role-relevant aggressive queries through …

abstract arxiv cs.cl dialogue evaluation improvement language language models large language large language models llm llms ordinary playing queries role roles struggle systems through training type