[Project] Making AMD GPUs competitive for LLM inference

There have been many LLM inference solutions since the bloom of open-source LLMs. Most of the performant inference solutions are based on CUDA and optimized for NVIDIA GPUs. In the meantime, with the high demand for compute availability, it is useful to bring support to a broader class of hardware accelerators. AMD is one potential candidate.

We build a project that makes it possible to compile LLMs and deploy them on AMD GPUs using ROCm and get competitive performance. More …

accelerators amd availability bloom compute cuda demand gpus hardware inference llm llms machinelearning making nvidia nvidia gpus project solutions support