[R] Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers

**Abstract:**

>Autoregressive Transformers adopted in Large Language Models (LLMs) are hard to scale to long sequences. Despite several works trying to reduce their computational cost, most of LLMs still adopt attention layers between all pairs of tokens in the sequence, thus incurring a quadratic cost. In this study, we present a novel approach that dynamically prunes contextual information while preserving the model's expressiveness, resulting in reduced memory and computational requirements during inference. Our method employs a learnable mechanism that determines …

abstract attention computational context cost dynamic language language models large language models llms machinelearning pruning reduce scale study tokens transformers