Making Attention Mechanisms More Robust and Interpretable with Virtual Adversarial Training. (arXiv:2104.08763v2 [cs.CL] UPDATED)

Adversarial training (AT) for attention mechanisms has successfully reduced
such drawbacks by considering adversarial perturbations. However, this
technique requires label information, and thus, its use is limited to
supervised settings. In this study, we explore the concept of incorporating
virtual AT (VAT) into the attention mechanisms, by which adversarial
perturbations can be computed even from unlabeled data. To realize this
approach, we propose two general training techniques, namely VAT for attention
mechanisms (Attention VAT) and "interpretable" VAT for attention mechanisms …

arxiv attention attention mechanisms making training virtual