Explaining Attention in Transformers [From The Encoder Point of View]

In this article, we will take a deep dive into the concept of attention in Transformer networks, particularly from the encoder’s perspective. We will cover the following topics:

• What is machine translation?
• Need for attention.
• How is attention computed using Recurrent Neural Networks (RNNs)?
• What is self-attention, and how is it computed using the Transformer’s encoder?
• Multi-headed attention in the Encoder.

Machine Translation

We will look at Neural machine translation (NMT) as a running …

attention multi-head attention nlp self-attention transformers