Serving Quantized LLMs on NVIDIA H100 Tensor Core GPUs

Quantization is a technique for making machine learning models smaller and faster. We quantize Llama2-70B-Chat, producing an equivalent-quality model that generates 2.2x more...

chat core faster generative-ai gpus h100 llama2 llms machine machine learning machine learning models making nvidia nvidia h100 quality quantization tensor tensor core gpus