HELM lite: Lightweight and Broad Capabilities Evaluation

It seems hard to believe that Holistic Evaluation of Language Models (HELM) was released only a year ago: November 2022 — ChatGPT had not even come out yet.  The original goal of HELM was to holistically evaluate all the language models we had access to on a set of representative scenarios (capturing language abilities, reasoning abilities, knowledge, etc.) and multiple metrics (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency).  As a result, we ended up with something that was conceptually elegant, …

capabilities chatgpt evaluation helm language language models set