How to Serve LLM Completions in Production

Preparations

To start, you need to compile llama.cpp. You can follow their README for instructions.

The server is compiled alongside other targets by default.

Once you have the server running, we can continue. We will use PHP Resonance framework.

Troubleshooting

Obtaining Open-Source LLM

I recommend starting either with llama2 or Mistral. You need to download the pretrained weights and convert them into GGUF format before they can be used with llama.cpp.

Starting Server Without a GPU

llama.cpp …

ai cpp framework llama llama2 llm mistral php production readme running serve server targets troubleshooting webdev will