A True Story of a Gradient that Vanished in an RNN

Why do they vanish and how to stop that from happening

Recurrent neural networks attempt to generalize feedforward neural networks to deal with sequence data. Their existence has revolutionized deep learning because they paved the way for numerous fascinating applications ranging from language modeling to translation that are much harder to achieve in classical machine learning. Nevertheless, one problem that prevents RNNs from prevailing in many RNN applications is their inability to deal with long-term dependences which naturally arise when …

backpropagation gradient nlp recurrent neural network rnn vanishing-gradient