A Data Cartography based MixUp for Pre-trained Language Models. (arXiv:2205.03403v1 [cs.CL])

Web: http://arxiv.org/abs/2205.03403

MixUp is a data augmentation strategy where additional samples are generated
during training by combining random pairs of training samples and their labels.
However, selecting random pairs is not potentially an optimal choice. In this
work, we propose TDMixUp, a novel MixUp strategy that leverages Training
Dynamics and allows more informative samples to be combined for generating new
data samples. Our proposed TDMixUp first measures confidence, variability,
(Swayamdipta et al., 2020), and Area Under the Margin (AUM) (Pleiss et al., …

arxiv data language language models models