[N] Language Processing Unit (LPU) makes inference of LLMs 10x faster

This week, a largely unknown company, **Groq**, demonstrated unprecedented speed running open-source LLMs such as Llama-2 (70 billion parameters) at more than 100 tokens per second, and Mixtral at nearly 500 tokens per second per user on a Groq’s Language Processing Unit (LPU).

For the **comparison**:

* “According to Groq, in similar tests, ChatGPT loads at 40-50 tokens per second, and Bard at 70 tokens per second on typical GPU-based computing systems.
* Context for 100 tokens per second per …

billion comparison faster groq inference language language processing llama llms machinelearning mixtral parameters per processing running speed tokens