[D] How are the popular LLM API servings optimized?

Currently there are a ton of offerings of various large langauge models hosted by companies like Together AI, Perplexity, Replit and many others.

They seem pretty fast especially for the 30B+ model sizes. Anyone know how these are optimized? Apart from the horizontal scaling across GPUs and probably dynamic batching (assuming the requests are large in number), what else are these companies doing?

Some of these companies also released the APIs the very next day the models come out - …

api batching companies dynamic gpus large langauge models llm machinelearning perplexity popular replit scaling together together ai