A Systematic Comparison of Syllogistic Reasoning in Humans and Language Models

arXiv:2311.00445v2 Announce Type: replace-cross
Abstract: A central component of rational behavior is logical inference: the process of determining which conclusions follow from a set of premises. Psychologists have documented several ways in which humans' inferences deviate from the rules of logic. Do language models, which are trained on text generated by humans, replicate such human biases, or are they able to overcome them? Focusing on the case of syllogisms -- inferences from two simple premises -- we show that, within …

abstract arxiv behavior comparison cs.ai cs.cl cs.lg humans inference inferences language language models logic process reasoning rules set text type