Evaluating Step-by-Step Reasoning through Symbolic Verification

arXiv:2212.08686v2 Announce Type: replace
Abstract: Pre-trained language models (LMs) have shown remarkable reasoning performance using explanations or chain-of-thoughts (CoT)) for in-context learning. On the other hand, these reasoning tasks are usually presumed to be more approachable for symbolic programming. To understand the mechanism of reasoning of LMs, we curate synthetic datasets containing equivalent (natural, symbolic) data pairs, where symbolic examples contain first-order logic rules and predicates from non-parametric knowledge bases (KBs), supporting automated verification of intermediate reasoning results. Then we …

abstract arxiv context cs.cl datasets in-context learning language language models lms natural performance programming reasoning step-by-step synthetic tasks thoughts through type verification