The Infamous Attention Mechanism in the Transformer architecture

The Infamous Attention Mechanism in the Transformer Architecture

THE WHY & WHEN?

It all started with a problem. How do you play around with sequential data?! People had the architecture to work with regression and classification problems, but sequential data was very different.

So, a new neural network architecture was introduced that had the concept of memory to enable working with sequential data. RNNs (LSTMs & GRUs).

RNNs & LSTMs did help …

architecture attention attention-mechanism classification data deep learning llm network network architecture neural network people regression transformer transformer architecture transformers work