Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective

arXiv:2310.11451v2 Announce Type: replace-cross
Abstract: Large Language Models (LLMs) inherently encode a wealth of knowledge within their parameters through pre-training on extensive corpora. While prior research has delved into operations on these parameters to manipulate the underlying implicit knowledge (encompassing detection, editing, and merging), there remains an ambiguous understanding regarding their transferability across models with varying scales. In this paper, we seek to empirically investigate knowledge transfer from larger to smaller models through a parametric perspective. To achieve this, we …

arxiv cs.ai cs.cl cs.lg knowledge language language models large language large language models parametric perspective transfer type