QuaCer-C: Quantitative Certification of Knowledge Comprehension in LLMs

arXiv:2402.15929v1 Announce Type: cross
Abstract: Large Language Models (LLMs) have demonstrated impressive performance on several benchmarks. However, traditional studies do not provide formal guarantees on the performance of LLMs. In this work, we propose a novel certification framework for LLM, QuaCer-C, wherein we formally certify the knowledge-comprehension capabilities of popular LLMs. Our certificates are quantitative - they consist of high-confidence, tight bounds on the probability that the target LLM gives the correct answer on any relevant knowledge comprehension prompt. Our …

abstract arxiv benchmarks capabilities certification cs.ai cs.cl cs.lg framework knowledge language language models large language large language models llm llms novel performance popular quantitative studies type work