FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets

arXiv:2307.10928v3 Announce Type: replace
Abstract: Evaluation of Large Language Models (LLMs) is challenging because instruction-following necessitates alignment with human values and the required set of skills varies depending on the instruction. However, previous studies have mainly focused on coarse-grained evaluation (i.e. overall preference-based evaluation), which limits interpretability since it does not consider the nature of user instructions that require instance-wise skill composition. In this paper, we introduce FLASK (Fine-grained Language Model Evaluation based on Alignment Skill Sets), a fine-grained evaluation …

alignment arxiv cs.ai cs.cl evaluation fine-grained flask language language model type