[D] How do you Serve and Scale Your LLMs in Production?

For people who have worked on self hosting your own models, I'm curious about your tech stack and architecture for model serving. Especially for those who are serving models larger than 30B. What optimizations and stack do you find effective for dealing with an environment where request volume is volatile (e.g can spike 10x in minutes), but responsiveness needs to be high?

architecture environment hosting llms machinelearning people production request scale serve stack tech tech stack