Positional Encoding Helps Recurrent Neural Networks Handle a Large Vocabulary

This study discusses the effects of positional encoding on recurrent neural networks (RNNs) utilizing synthetic benchmarks. Positional encoding "time-stamps" data points in time series and complements the capabilities of Transformer neural networks, which lack an inherent mechanism for representing the data order. By contrast, RNNs can encode the temporal information of data points on their own, rendering their use of positional encoding seemingly "redundant". Nonetheless, empirical investigations reveal the effectiveness of positional encoding even when coupled with RNNs, specifically for …

benchmarks capabilities contrast cs.lg cs.ne data effects encode encoding information networks neural networks positional encoding recurrent neural networks series study synthetic temporal time series transformer