NumeroLogic: Number Encoding for Enhanced LLMs' Numerical Reasoning

arXiv:2404.00459v1 Announce Type: new
Abstract: Language models struggle with handling numerical data and performing arithmetic operations. We hypothesize that this limitation can be partially attributed to non-intuitive textual numbers representation. When a digit is read or generated by a causal language model it does not know its place value (e.g. thousands vs. hundreds) until the entire number is processed. To address this issue, we propose a simple adjustment to how numbers are represented by including the count of digits before …

abstract arxiv causal cs.cl data digit encoding generated language language model language models llms numbers numerical operations reasoning representation struggle textual type value