Deploying LLMs Into Production Using TensorRT LLM

A guide on accelerating inference performance

Image by author — Created using Stable Diffusion XL

Intro

Open-source large language models have lived up to the hype. Many companies that use GPT-3.5 or GPT-4 in production have realized that these models are simply not scalable from a cost perspective. Because of this, enterprises are looking for good open-source alternatives. Recent models like Mixtral and Llama 2 have shown stellar results when it comes to output quality. But, scaling these models to …

hands-on-tutorials large language models machine learning software development tensorrt