[R] Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers

**Paper**: [https://arxiv.org/abs/2311.10642](https://arxiv.org/abs/2311.10642)

**Code**: [https://github.com/vulus98/Rethinking-attention](https://github.com/vulus98/Rethinking-attention)

**Abstract**:

>This work presents an analysis of the effectiveness of using standard shallow feed-forward networks to mimic the behavior of the attention mechanism in the original Transformer model, a state-of-the-art architecture for sequence-to-sequence tasks. We substitute key elements of the attention mechanism in the Transformer with simple feed-forward networks, trained using the original components via knowledge distillation. Our experiments, conducted on the IWSLT2017 dataset, reveal the capacity of these "attentionless Transformers" to rival the performance of …

abstract analysis architecture art attention behavior components distillation knowledge machinelearning networks simple standard state tasks transformer transformer model work