[R] The Expressive Power of Transformers with Chain of Thought

[Paper](https://arxiv.org/abs/2310.07923). I am not affiliated with the authors.

Abstract:

>Recent theoretical work has identified surprisingly simple reasoning problems, such as checking if two nodes in a graph are connected or simulating finite-state machines, that are provably unsolvable by standard transformers that answer immediately after reading their input. However, in practice, transformers' reasoning can be improved by allowing them to use a "chain of thought" or "scratchpad", i.e., generate and condition on a sequence of intermediate tokens before answering. Motivated by …

abstract chain of thought generate graph machinelearning machines practice reading reasoning simple standard state them thought transformers work