On the Representational Capacity of Recurrent Neural Language Models

arXiv:2310.12942v4 Announce Type: replace-cross
Abstract: This work investigates the computational expressivity of language models (LMs) based on recurrent neural networks (RNNs). Siegelmann and Sontag (1992) famously showed that RNNs with rational weights and hidden states and unbounded computation time are Turing complete. However, LMs define weightings over strings in addition to just (unweighted) language membership and the analysis of the computational power of RNN LMs (RLMs) should reflect this. We extend the Turing completeness result to the probabilistic case, showing …

abstract arxiv capacity computation computational cs.cl cs.lg hidden however language language models lms networks neural networks recurrent neural networks strings turing type work