Exploring the Latency/Throughput & Cost Space for LLM Inference // Timothée Lacroix // LLM 3 Talk 3

// Abstract
Getting the right LLM inference stack means choosing the right model for your task, and running it on the right hardware, with proper inference code. This talk will go through popular inference stacks and set-ups, detailing what makes inference costly. We'll talk about the current generation of open-source models and how to make the best use of them, but we will also touch on features currently missing from the open-source serving stack as well as what the future …

abstract code cost hardware inference latency llm popular running set space stack stacks talk through ups