Transformers discover an elementary calculation system exploiting local attention and grid-like problem representation. (arXiv:2207.02536v1 [cs.LG])

Mathematical reasoning is one of the most impressive achievements of human
intellect but remains a formidable challenge for artificial intelligence
systems. In this work we explore whether modern deep learning architectures can
learn to solve a symbolic addition task by discovering effective arithmetic
procedures. Although the problem might seem trivial at first glance,
generalizing arithmetic knowledge to operations involving a higher number of
terms, possibly composed by longer sequences of digits, has proven extremely
challenging for neural networks. Here we …

arxiv attention elementary lg local attention representation transformers