Learning Transductions and Alignments with RNN Seq2seq Models

arXiv:2303.06841v4 Announce Type: replace
Abstract: The paper studies the capabilities of Recurrent-Neural-Network sequence to sequence (RNN seq2seq) models in learning four transduction tasks: identity, reversal, total reduplication, and quadratic copying. These transductions are traditionally well studied under finite state transducers and attributed with increasing complexity. We find that RNN seq2seq models are only able to approximate a mapping that fits the training or in-distribution data, instead of learning the underlying functions. Although attention makes learning more efficient and robust, it …

abstract arxiv capabilities complexity cs.cl identity network paper rnn seq2seq sequence to sequence state studies tasks total type