Measuring the Robustness of NLP Models to Domain Shifts

arXiv:2306.00168v5 Announce Type: replace
Abstract: Existing research on Domain Robustness (DR) suffers from disparate setups, limited task variety, and scarce research on recent capabilities such as in-context learning. Furthermore, the common practice of measuring DR might not be fully accurate. Current research focuses on challenge sets and relies solely on the Source Drop (SD): Using the source in-domain performance as a reference point for degradation. However, we argue that the Target Drop (TD), which measures degradation from the target in-domain …

abstract arxiv capabilities challenge context cs.cl current domain in-context learning measuring nlp nlp models practice research robustness type