Syn-QA2: Evaluating False Assumptions in Long-tail Questions with Synthetic QA Datasets

arXiv:2403.12145v1 Announce Type: new
Abstract: Sensitivity to false assumptions (or false premises) in information-seeking questions is critical for robust question-answering (QA) systems. Recent work has shown that false assumptions in naturally occurring questions pose challenges to current models, with low performance on both generative QA and simple detection tasks (Kim et al. 2023). However, the focus of existing work on naturally occurring questions leads to a gap in the analysis of model behavior on the long tail of the distribution …

abstract arxiv assumptions challenges cs.cl current datasets detection false generative information low performance question questions robust sensitivity simple syn synthetic systems tasks type work