A Theoretical Result on the Inductive Bias of RNN Language Models

arXiv:2402.15814v1 Announce Type: cross
Abstract: Recent work by Hewitt et al. (2020) provides a possible interpretation of the empirical success of recurrent neural networks (RNNs) as language models (LMs).
It shows that RNNs can efficiently represent bounded hierarchical structures that are prevalent in human language.
This suggests that RNNs' success might be linked to their ability to model hierarchy.
However, a closer inspection of Hewitt et al.'s (2020) construction shows that it is not limited to hierarchical LMs, posing the …

abstract arxiv bias cs.cc cs.cl cs.lg hierarchical human inductive interpretation language language models lms networks neural networks recurrent neural networks rnn shows success type work