[D] How does fast inference work with state of the art LLMs?

I’ve read that inference speed for models like Llama-2 70B is ~10 t/s at best. So that left me wondering how the extremely large models like GPT-4 (1T params?) do their fast 20 t/s inference. With 10x the params, they gotta have at least 3x the layers(?) So that should make its inference much slower. Am I missing anything? What kind of further improvements might these companies be doing to power their fast APIs?

Edit: I must mention that you …

70b art gpt gpt-4 inference large models least llama llms machinelearning params speed state state of the art work