[D] How is it that the latency to decode 1 new token with an LLM is constant independent of total sequence length, when caching KV?

I’m struggling to understand something about [transformer inference arithmetic](https://kipp.ly/transformer-inference-arithmetic/) with KV caching, together with some benchmarking results.

**How is it that the latency to decode 1 new token is constant independent of total sequence length (input+output)?**

Let’s assume batch size 1, and simple multi head attention. At each step t, even though we save recomputing KV for the entire sequence, we do have to compute attention using the current input’s Q against a growing kv cache, which represents more FLOPS …

attention cache compute current decode head independent latency machinelearning save simple token total