Evaluating Large Language Models Using Contrast Sets: An Experimental Approach

arXiv:2404.01569v1 Announce Type: cross
Abstract: In the domain of Natural Language Inference (NLI), especially in tasks involving the classification of multiple input texts, the Cross-Entropy Loss metric is widely employed as a standard for error measurement. However, this metric falls short in effectively evaluating a model's capacity to understand language entailments. In this study, we introduce an innovative technique for generating a contrast set for the Stanford Natural Language Inference (SNLI) dataset. Our strategy involves the automated substitution of verbs, …

abstract arxiv capacity classification contrast cross-entropy cs.ai cs.cl cs.lg domain entropy error experimental however inference language language models large language large language models loss measurement multiple natural natural language standard tasks type