[P] Paddler (stateful load balancer custom-tailored for llama.cpp)

I have started this project recently. It allows us to self-host llama.cpp and use it with open-source models.

It started to gain some traction recently, and it is production-ready.

It allows scaling from zero instances, so if you are using cloud providers to prototype your ideas with open-source LLMs, you will only pay for what you actually use. If there is a period of inactivity, you can use it to shut down expensive GPU instances and only leave some cheap …

cloud cloud providers cpp custom ideas instances llama llms machinelearning open-source models production project prototype scaling will you