How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Study

arXiv:2402.16061v1 Announce Type: new
Abstract: Previous work has showcased the intriguing capability of large language models (LLMs) in retrieving facts and processing context knowledge. However, only limited research exists on the layer-wise capability of LLMs to encode knowledge, which challenges our understanding of their internal mechanisms. In this paper, we devote the first attempt to investigate the layer-wise capability of LLMs through probing tasks. We leverage the powerful generative capability of ChatGPT to construct probing datasets, providing diverse and coherent …

arxiv context cs.cl encode knowledge language language models large language large language models layer study type wise