Do Efficient Transformers Really Save Computation?

arXiv:2402.13934v1 Announce Type: new
Abstract: As transformer-based language models are trained on increasingly large datasets and with vast numbers of parameters, finding more efficient alternatives to the standard Transformer has become very valuable. While many efficient Transformers and Transformer alternatives have been proposed, none provide theoretical guarantees that they are a suitable replacement for the standard Transformer. This makes it challenging to identify when to use a specific model and what directions to prioritize for further investigation. In this paper, …

abstract arxiv become computation cs.ai cs.cl cs.lg datasets language language models large datasets numbers parameters replacement save standard stat.ml transformer transformers type vast