Harmonic LLMs are Trustworthy

arXiv:2404.19708v1 Announce Type: new
Abstract: We introduce an intuitive method to test the robustness (stability and explainability) of any black-box LLM in real-time, based upon the local deviation from harmoniticity, denoted as $\gamma$. To the best of our knowledge this is the first completely model-agnostic and unsupervised method of measuring the robustness of any given response from an LLM, based upon the model itself conforming to a purely mathematical standard. We conduct human annotation experiments to show the positive correlation …

abstract arxiv best of box cs.ai cs.cl cs.hc cs.lg deviation explainability knowledge llm llms measuring model-agnostic real-time robustness stability test trustworthy type unsupervised