CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models

arXiv:2404.13161v1 Announce Type: cross
Abstract: Large language models (LLMs) introduce new security risks, but there are few comprehensive evaluation suites to measure and reduce these risks. We present BenchmarkName, a novel benchmark to quantify LLM security risks and capabilities. We introduce two new areas for testing: prompt injection and code interpreter abuse. We evaluated multiple state-of-the-art (SOTA) LLMs, including GPT-4, Mistral, Meta Llama 3 70B-Instruct, and Code Llama. Our results show that conditioning away risk of attack remains an unsolved …

abstract arxiv benchmark capabilities cs.cr cs.lg cybersecurity evaluation language language models large language large language models llm llms llm security novel prompt prompt injection reduce risks security testing type