Why You Should Not Use Numeric Evals For LLM As a Judge

Image created by author using Dall-E 3

Testing major LLMs on how well they conduct numeric evaluations

In addition to generating text for a growing number of industry applications, LLMs are now widely being used as evaluation tools. Models quantify the relevance of retrieved documents in retrieval systems, gauge the sentiment of comments and posts, and more — evaluating both human and AI-generated text. These evaluations are often either numeric or categorical.

Different types of LLM evals (diagram by author) …

applications author dall dall-e documents evals evaluation evaluation-metric image industry judge llm llm-evaluation llmops llms major mixtral 8x7b retrieval systems text thoughts-and-theory tools