Paddler - open-source llama.cpp load balancer (self-host LLMs in production)

Paddler is an open-source load balancer and reverse proxy designed to optimize servers running llama.cpp.

Typical strategies like round robin or least connections are not effective for llama.cpp servers, which need slots for continuous batching and concurrent requests.

Paddler overcomes this by maintaining a stateful load balancer that is aware of each server's available slots, ensuring efficient request distribution. Additionally, Paddler uses agents to monitor the health of individual llama.cpp instances, providing feedback to the load balancer for optimal …

ai batching continuous cpp devops least llama llms opensource production requests running servers strategies