Arithmetic with Language Models: from Memorization to Computation

arXiv:2308.01154v3 Announce Type: replace-cross
Abstract: A better understanding of the emergent computation and problem-solving capabilities of recent large language models is of paramount importance to further improve them and broaden their applicability. This work investigates how a language model, trained to predict the next token, can perform arithmetic computations generalizing beyond training data. Binary addition and multiplication constitute a good testbed for this purpose, since they require a very small vocabulary and exhibit relevant input/output discontinuities making smooth input interpolation …

abstract arxiv beyond capabilities computation cs.ai cs.cl importance language language model language models large language large language models next problem-solving them token training type understanding work