CLI tool to benchmark 100+LLMs response, response time, cost

Hi r/LanguageTechnology ,I built a CLI tool to benchmark 100+ LLMs for a given question. Benchmark output allows you to compare responses, response time and cost.

Try it here: [https://github.com/BerriAI/litellm/blob/main/cookbook/benchmark/readme.md](https://github.com/BerriAI/litellm/blob/main/cookbook/benchmark/readme.md)

**CLI Output:**

|Model|Response|Response Time|Cost|
|:-|:-|:-|:-|
|gpt-3.5|As an AI language model, I cannot provide up-to-date information or predict|2.1|$0.000122|
|claude-2|I'm not able to provide information about future IPO plans or dates for BerriAI|3.2|$0.0010142|
|llama2|I do not have any information about when or if BerriAI will have an initial|0.01|$0.000522|

Simply select your LLMs, …

ai language model benchmark claude cli cost future gpt gpt-3 gpt-3.5 information language language model languagetechnology llms responses tool