R-Judge: Benchmarking Safety Risk Awareness for LLM Agents

arXiv:2401.10019v2 Announce Type: replace
Abstract: Large language models (LLMs) have exhibited great potential in autonomously completing tasks across real-world applications. Despite this, these LLM agents introduce unexpected safety risks when operating in interactive environments. Instead of centering on LLM-generated content safety in most prior studies, this work addresses the imperative need for benchmarking the behavioral safety of LLM agents within diverse environments. We introduce R-Judge, a benchmark crafted to evaluate the proficiency of LLMs in judging and identifying safety risks …

agents arxiv benchmarking cs.ai cs.cl judge llm risk safety type