[D] How do large companies get their LLMs to give sub second responses?

Curious how companies like Google, MSFT, etc are able to have their LLMs and ML models have very fast responses. Do they just have crazy powerful gpus or split inference amongst gpus.

companies etc google gpus inference llms machinelearning ml models msft responses