Taxonomy-based CheckList for Large Language Model Evaluation

arXiv:2402.10899v1 Announce Type: new
Abstract: As large language models (LLMs) have been used in many downstream tasks, the internal stereotypical representation may affect the fairness of the outputs. In this work, we introduce human knowledge into natural language interventions and study pre-trained language models' (LMs) behaviors within the context of gender bias. Inspired by CheckList behavioral testing, we present a checklist-style task that aims to probe and quantify LMs' unethical behaviors through question-answering (QA). We design three comparison studies to …

abstract arxiv bias checklist context cs.cl evaluation fairness gender gender bias human knowledge language language model language models large language large language model large language models llms lms natural natural language representation study tasks taxonomy type work