Results from Deploying Quantized version of SOLAR 10.7B-Instruct

Hello everyone,

Been working on optimizing upstart.ai SOLAR-10.7B-Instruct-v1.0 model and wanted to share our insights:

🚀 **Our Approach:** Quantized the model using Auto-GPTQ, then deployed with vLLM.

Results: In a serverless setup, we saw 1.37 sec inference, 111.54 tokens/sec, and an 11.69 sec cold start on Nvidia A100 GPU.

https://preview.redd.it/eyym3rc3zlac1.png?width=1600&format=png&auto=webp&s=5846a8b2eb4cf6d9cd8f12545d498c37d3653056

Other Methods Tested: Although Auto-GPTQ was an option, our experience suggests that vLLM is the superior choice for deployment.

Looking forward to hearing about your experiences with similar projects!

a100 a100 gpu auto cold start gpu hello inference insights machinelearningnews nvidia nvidia a100 nvidia a100 gpu sec serverless setup solar tokens