Crafting In-context Examples according to LMs' Parametric Knowledge

arXiv:2311.09579v2 Announce Type: replace
Abstract: In-context learning can improve the performances of knowledge-rich tasks such as question answering. In such scenarios, in-context examples trigger a language model (LM) to surface information stored in its parametric knowledge. We study how to better construct in-context example sets, based on whether the model is aware of the in-context examples. We identify 'known' examples, where models can correctly answer from their parametric knowledge, and 'unknown' ones. Our experiments show that prompting with 'unknown' examples …

abstract arxiv construct context cs.cl example examples in-context learning information knowledge language language model lms parametric performances question question answering study surface tasks type