Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark

arXiv:2402.12243v1 Announce Type: new
Abstract: Text-to-SQL, which involves translating natural language into Structured Query Language (SQL), is crucial for enabling broad access to structured databases without expert knowledge. However, designing models for such tasks is challenging due to numerous factors, including the presence of 'noise,' such as ambiguous questions and syntactical errors. This study provides an in-depth analysis of the distribution and types of noise in the widely used BIRD-Bench benchmark and the impact of noise on models. While BIRD-Bench …

abstract arxiv benchmark bird cs.cl databases designing effects enabling expert knowledge language natural natural language noise query sql structured query language tasks text text-to-sql type understanding