xLSTM: Extended Long Short-Term Memory

arXiv:2405.04517v1 Announce Type: new
Abstract: In the 1990s, the constant error carousel and gating were introduced as the central ideas of the Long Short-Term Memory (LSTM). Since then, LSTMs have stood the test of time and contributed to numerous deep learning success stories, in particular they constituted the first Large Language Models (LLMs). However, the advent of the Transformer technology with parallelizable self-attention at its core marked the dawn of a new era, outpacing LSTMs at scale. We now raise …

abstract arxiv contributed cs.ai cs.lg deep learning error however ideas language language models large language large language models llms long short-term memory lstm memory stat.ml stories success success stories test type