[R] A Primer on the Inner Workings of Transformer-based Language Models

**Authors**: Javier Ferrando (UPC), Gabriele Sarti (RUG), Arianna Bisazza (RUG), Marta Costa-jussà (Meta)

**Paper:** [https://arxiv.org/abs/2405.00208](https://arxiv.org/abs/2405.00208)

**Abstract:**

>The rapid progress of research aimed at interpreting the inner workings of advanced language models has highlighted a need for contextualizing the insights gained from years of work in this area. This primer provides a concise technical introduction to the current techniques used to interpret the inner workings of Transformer-based language models, focusing on the generative decoder-only architecture. We conclude by presenting a comprehensive …

abstract advanced authors insights language language models machinelearning meta primer progress research transformer work