Mixture-of-Depths: Dynamically allocating compute in transformer-based language models

arXiv:2404.02258v1 Announce Type: new
Abstract: Transformer-based language models spread FLOPs uniformly across input sequences. In this work we demonstrate that transformers can instead learn to dynamically allocate FLOPs (or compute) to specific positions in a sequence, optimising the allocation along the sequence for different layers across the model depth. Our method enforces a total compute budget by capping the number of tokens ($k$) that can participate in the self-attention and MLP computations at a given layer. The tokens to be …

abstract arxiv compute cs.cl cs.lg language language models learn transformer transformers type work