Hyena Hierarchy: Towards Larger Convolutional Language Models

Recent advances in deep learning have relied heavily on the use of large
Transformers due to their ability to learn at scale. However, the core
building block of Transformers, the attention operator, exhibits quadratic
cost in sequence length, limiting the amount of context accessible.

attention building context core cost deep learning language language models learn research scale transformers