Data Diversity Matters for Robust Instruction Tuning

Recent works have shown that by curating high quality and diverse instruction tuning datasets, we can significantly improve instruction-following capabilities. However, creating such datasets is difficult and most works rely on manual curation or proprietary language models. Automatic data curation is difficult as it is still not clear how we can define diversity for instruction tuning, how diversity and quality depend on one other, and how we can optimize dataset quality and diversity. To resolve these issue, we propose a …

capabilities clear cs.cl cs.lg curation data data curation data diversity datasets diverse diversity language language models proprietary quality robust