Improving the Numerical Reasoning Skills of Pretrained Language Models. (arXiv:2205.06733v1 [cs.CL])

State-of-the-art pretrained language models tend to perform below their
capabilities when applied out-of-the-box on tasks that require reasoning over
numbers. Recent work sees two main reasons for this: (1) popular tokenisation
algorithms are optimized for common words, and therefore have limited
expressiveness for numbers, and (2) common pretraining objectives do not target
numerical reasoning or understanding numbers at all. Recent approaches usually
address them separately and mostly by proposing architectural changes or
pretraining models from scratch. In this paper, we …

arxiv language language models numerical reasoning skills