[D] Serverless Inference for Llama2

Serverless Inference for Llama2

I am part of a small (startup like) organization and want to use a model to answer client requests but these should not be 24/7 so I started looking at serverless inference. I have been warned about cold start times since the desired latency is of about 1-5 sec. I am using a Llama2-7b-GPTQ model (quantized) and also experimenting with the 13b version. The model weights take about 10GB of memory. I still do not have …

client cold start inference latency llama2 machinelearning organization part sec serverless serverless inference small startup