[D] Can somebody help check my math to see if I'm understanding Microsoft's Retentive Network paper correctly? I'm confused how we are enriching the tokens with enough context.

Relevant Paper: [2307.08621.pdf (arxiv.org)](https://arxiv.org/pdf/2307.08621.pdf)

So the definition of the recurrent representation of the retention mechanism is below

>Sn = γSn−1 + K⊺nVn
>
>Retention(Xn) = QnSn, n = 1, · · · , |x|

γ is a decay factor, and K, Q, and V have their standard transformer definitions.

What confuses me is the derivation of Sn. The formula makes it look like a scalar. But if that's the case, are we saying that for a given token, the retention …

definition definitions derivation look machinelearning representation retention standard transformer