Benchmarking the Text-to-SQL Capability of Large Language Models: A Comprehensive Evaluation

arXiv:2403.02951v1 Announce Type: new
Abstract: Large Language Models (LLMs) have emerged as a powerful tool in advancing the Text-to-SQL task, significantly outperforming traditional methods. Nevertheless, as a nascent research field, there is still no consensus on the optimal prompt templates and design frameworks. Additionally, existing benchmarks inadequately explore the performance of LLMs across the various sub-tasks of the Text-to-SQL process, which hinders the assessment of LLMs' cognitive capabilities and the optimization of LLM-based solutions.To address the aforementioned issues, we firstly …

abstract arxiv benchmarking benchmarks capability consensus cs.ai cs.cl design evaluation explore frameworks language language models large language large language models llms prompt research sql text text-to-sql tool type