Towards Unbiased Evaluation of Detecting Unanswerable Questions in EHRSQL

arXiv:2405.01588v1 Announce Type: new
Abstract: Incorporating unanswerable questions into EHR QA systems is crucial for testing the trustworthiness of a system, as providing non-existent responses can mislead doctors in their diagnoses. The EHRSQL dataset stands out as a promising benchmark because it is the only dataset that incorporates unanswerable questions in the EHR QA system alongside practical questions. However, in this work, we identify a data bias in these unanswerable questions; they can often be discerned simply by filtering with …

abstract arxiv benchmark cs.ai cs.cl dataset doctors ehr evaluation questions responses systems testing type unbiased