Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning. (arXiv:2205.09712v1 [cs.AI])

Large language models (LLMs) have been shown to be capable of impressive
few-shot generalisation to new tasks. However, they still tend to perform
poorly on multi-step logical reasoning problems. Here we carry out a
comprehensive evaluation of LLMs on 50 tasks that probe different aspects of
logical reasoning. We show that language models tend to perform fairly well at
single step inference or entailment tasks, but struggle to chain together
multiple reasoning steps to solve more complex problems. In light …

ai arxiv inference language language models large language models reasoning