Can Large Language Models Follow Concept Annotation Guidelines? A Case Study on Scientific and Financial Domains

arXiv:2311.08704v2 Announce Type: replace
Abstract: Although large language models (LLMs) exhibit remarkable capacity to leverage in-context demonstrations, it is still unclear to what extent they can learn new concepts or facts from ground-truth labels. To address this question, we examine the capacity of instruction-tuned LLMs to follow in-context concept guidelines for sentence labeling tasks. We design guidelines that present different types of factual and counterfactual concept definitions, which are used as prompts for zero-shot sentence classification tasks. Our results show …

abstract annotation arxiv capacity case case study concept concepts context cs.ai cs.cl domains facts financial ground-truth guidelines instruction-tuned labels language language models large language large language models learn llms question replace scientific study truth type