Principles from Clinical Research for NLP Model Generalization

arXiv:2311.03663v3 Announce Type: replace
Abstract: The NLP community typically relies on performance of a model on a held-out test set to assess generalization. Performance drops observed in datasets outside of official test sets are generally attributed to "out-of-distribution" effects. Here, we explore the foundations of generalizability and study the factors that affect it, articulating lessons from clinical studies. In clinical research, generalizability is an act of reasoning that depends on (a) internal validity of experiments to ensure controlled measurement of …

abstract arxiv clinical clinical research community cs.cl datasets distribution effects explore model generalization nlp nlp model performance research set study test type