How often are errors in natural language reasoning due to paraphrastic variability?

arXiv:2404.11717v1 Announce Type: new
Abstract: Large language models have been shown to behave inconsistently in response to meaning-preserving paraphrastic inputs. At the same time, researchers evaluate the knowledge and reasoning abilities of these models with test evaluations that do not disaggregate the effect of paraphrastic variability on performance. We propose a metric for evaluating the paraphrastic consistency of natural language reasoning models based on the probability of a model achieving the same correctness on two paraphrases of the same problem. …

abstract arxiv cs.cl errors inputs knowledge language language models large language large language models meaning natural natural language performance reasoning researchers test type