The Expressive Power of Transformers with Chain of Thought

arXiv:2310.07923v4 Announce Type: replace
Abstract: Recent theoretical work has identified surprisingly simple reasoning problems, such as checking if two nodes in a graph are connected or simulating finite-state machines, that are provably unsolvable by standard transformers that answer immediately after reading their input. However, in practice, transformers' reasoning can be improved by allowing them to use a "chain of thought" or "scratchpad", i.e., generate and condition on a sequence of intermediate tokens before answering. Motivated by this, we ask: Does …

abstract arxiv chain of thought cs.cc cs.cl cs.lg cs.lo graph however machines nodes power practice reading reasoning simple standard state thought transformers type work