Serving fastchat on single GPU and 5 models!

I tired playing around with open tools to serve multiple models on a Single A100 80GB

I was able to squeeze in 5 language models using VLLM, served via fastchat.

​

Here is a public instance for 72 hrs - Pls note that the battle mode is broken, other tabs work.

[https://c8168701070daa5bf3.gradio.live/](https://c8168701070daa5bf3.gradio.live/)

​

Llama 3 BB (8K context)
Gemma 8B (8k Context)
Phi 3 128K (18K context)
DeciLM 7B (8K Context)
Stable LM 1.6B (4k Context)

​

Is it stupid? …

a100 deeplearning gpu instance language language models multiple playing public serve tools via work