Evidence from counterfactual tasks supports emergent analogical reasoning in large language models

arXiv:2404.13070v1 Announce Type: new
Abstract: We recently reported evidence that large language models are capable of solving a wide range of text-based analogy problems in a zero-shot manner, indicating the presence of an emergent capacity for analogical reasoning. Two recent commentaries have challenged these results, citing evidence from so-called `counterfactual' tasks in which the standard sequence of the alphabet is arbitrarily permuted so as to decrease similarity with materials that may have been present in the language model's training data. …

abstract analogy arxiv capacity counterfactual cs.ai cs.cl evidence language language models large language large language models reasoning results tasks text type zero-shot