Retentive or Forgetful? Diving into the Knowledge Memorizing Mechanism of Language Models

arXiv:2305.09144v2 Announce Type: replace
Abstract: Memory is one of the most essential cognitive functions serving as a repository of world knowledge and episodes of activities. In recent years, large-scale pre-trained language models have shown remarkable memorizing ability. On the contrary, vanilla neural networks without pre-training have been long observed suffering from the catastrophic forgetting problem. To investigate such a retentive-forgetful contradiction and understand the memory mechanism of language models, we conduct thorough experiments by controlling the target knowledge types, the …

abstract arxiv cognitive cs.ai cs.cl episodes functions knowledge language language models memory networks neural networks pre-training scale training type world