ReLU and Addition-based Gated RNN. (arXiv:2308.05629v1 [cs.LG])

We replace the multiplication and sigmoid function of the conventional
recurrent gate with addition and ReLU activation. This mechanism is designed to
maintain long-term memory for sequence processing but at a reduced
computational cost, thereby opening up for more efficient execution or larger
models on restricted hardware. Recurrent Neural Networks (RNNs) with gating
mechanisms such as LSTM and GRU have been widely successful in learning from
sequential data due to their ability to capture long-term dependencies.
Conventionally, the update based …

arxiv computational cost function hardware long-term memory networks neural networks processing recurrent neural networks relu rnn sigmoid