LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Models

arXiv:2402.10524v1 Announce Type: cross
Abstract: Automatic side-by-side evaluation has emerged as a promising approach to evaluating the quality of responses from large language models (LLMs). However, analyzing the results from this evaluation approach raises scalability and interpretability challenges. In this paper, we present LLM Comparator, a novel visual analytics tool for interactively analyzing results from automatic side-by-side evaluation. The tool supports interactive workflows for users to understand when and why a model performs better or worse than a baseline model, …

abstract analytics arxiv challenges cs.ai cs.cl cs.hc cs.lg evaluation interpretability language language models large language large language models llm llms novel paper quality raises responses scalability type visual visual analytics