FOLIO: Natural Language Reasoning with First-Order Logic

arXiv:2209.00840v2 Announce Type: replace
Abstract: Large language models (LLMs) have achieved remarkable performance on a variety of natural language understanding tasks. However, existing benchmarks are inadequate in measuring the complex logical reasoning capabilities of a model. We present FOLIO, a human-annotated, logically complex and diverse dataset for reasoning in natural language (NL), equipped with first-order logic (FOL) annotations. FOLIO consists of 1,430 examples (unique conclusions), each paired with one of 487 sets of premises used to deductively reason for the …

abstract arxiv benchmarks capabilities cs.cl dataset diverse however human language language models language understanding large language large language models llms logic measuring natural natural language performance reasoning replace tasks type understanding