Next-Gen AI: RecurrentGemma (Long Context Length)

A brand new Language Model Architecture: RecurrentLLM. Moving Past Transformers.
Google developed RecurrentGemma-2B and compares this new LM architecture (!) with the classical transformer based, quadratic complexity of a self-attention Gemma 2B. And the new throughput is: about 6000 tokens per second.

Introduction and Model Architecture:
The original paper by Google introduces "RecurrentGemma-2B," leveraging the Griffin architecture, which moves away from traditional global attention mechanisms in favor of a combination of linear recurrences and local attention. This design enables the …

architecture attention brand complexity context gemma gen gen ai google introduction language language model moving next next-gen paper per self-attention tokens transformer transformers