Meta-Cognitive Analysis: Evaluating Declarative and Procedural Knowledge in Datasets and Large Language Models

arXiv:2403.09750v1 Announce Type: new
Abstract: Declarative knowledge and procedural knowledge are two key parts in meta-cognitive theory, and these two hold significant importance in pre-training and inference of LLMs. However, a comprehensive analysis comparing these two types of knowledge is lacking, primarily due to challenges in definition, probing and quantitative assessment. In this paper, we explore from a new perspective by providing ground-truth knowledge for LLMs and evaluating the effective score. Through extensive experiments with widely-used datasets and models, we …

abstract analysis arxiv challenges cognitive cs.ai cs.cl datasets definition however importance inference key knowledge language language models large language large language models llms meta pre-training theory training type types