Extreme Miscalibration and the Illusion of Adversarial Robustness

arXiv:2402.17509v1 Announce Type: new
Abstract: Deep learning-based Natural Language Processing (NLP) models are vulnerable to adversarial attacks, where small perturbations can cause a model to misclassify. Adversarial Training (AT) is often used to increase model robustness. However, we have discovered an intriguing phenomenon: deliberately or accidentally miscalibrating models masks gradients in a way that interferes with adversarial attack search methods, giving rise to an apparent increase in robustness. We show that this observed gain in robustness is an illusion of …

abstract adversarial adversarial attacks adversarial training arxiv attacks cs.cl deep learning language language processing masks model robustness natural natural language natural language processing nlp processing robustness small training type vulnerable