EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria

arXiv:2309.13633v2 Announce Type: replace-cross
Abstract: By simply composing prompts, developers can prototype novel generative applications with Large Language Models (LLMs). To refine prototypes into products, however, developers must iteratively revise prompts by evaluating outputs to diagnose weaknesses. Formative interviews (N=8) revealed that developers invest significant effort in manually evaluating outputs as they assess context-specific and subjective criteria. We present EvalLM, an interactive system for iteratively refining prompts by evaluating multiple outputs on user-defined criteria. By describing criteria in natural language, …

abstract applications arxiv cs.ai cs.cl cs.hc developers evaluation generative interactive interviews language language model language models large language large language model large language models llms novel products prompts refine type