Using Counterfactual Tasks to Evaluate the Generality of Analogical Reasoning in Large Language Models

arXiv:2402.08955v1 Announce Type: cross
Abstract: Large language models (LLMs) have performed well on several reasoning benchmarks, including ones that test analogical reasoning abilities. However, it has been debated whether they are actually performing humanlike abstract reasoning or instead employing less general processes that rely on similarity to what has been seen in their training data. Here we investigate the generality of analogy-making abilities previously claimed for LLMs (Webb, Holyoak, & Lu, 2023). We take one set of analogy problems used …

abstract arxiv benchmarks counterfactual cs.ai cs.cl general humanlike language language models large language large language models llms processes reasoning tasks test type