INFINI Attention explained: 1 Mio Context Length

New Infini attention transformer designed by Google for a context length of 1 mio token.

Infini-attention integrates a compressive memory component within the vanilla attention mechanism. This integration allows the model to handle very long input sequences by storing older attention key-value (KV) states in a compressive memory format instead of discarding them, which is the typical approach. These states can then be retrieved using attention queries for subsequent inputs, effectively allowing the model to "remember" and utilize an extensive …

attention context explained format google integration key memory them token transformer value