Learning the greatest common divisor: explaining transformer predictions

arXiv:2308.15594v2 Announce Type: replace
Abstract: The predictions of small transformers, trained to calculate the greatest common divisor (GCD) of two positive integers, can be fully characterized by looking at model inputs and outputs. As training proceeds, the model learns a list $\mathcal D$ of integers, products of divisors of the base used to represent integers and small primes, and predicts the largest element of $\mathcal D$ that divides both inputs. Training distributions impact performance. Models trained from uniform operands only …

abstract arxiv cs.ai cs.lg inputs integers list positive predictions products small training transformer transformers type