Alleviating the Inequality of Attention Heads for Neural Machine Translation. (arXiv:2009.09672v2 [cs.CL] UPDATED)

Recent studies show that the attention heads in Transformer are not equal. We
relate this phenomenon to the imbalance training of multi-head attention and
the model dependence on specific heads. To tackle this problem, we propose a
simple masking method: HeadMask, in two specific ways. Experiments show that
translation improvements are achieved on multiple language pairs. Subsequent
empirical analyses also support our assumption and confirm the effectiveness of
the method.

arxiv attention inequality machine machine translation neural machine translation translation