Kformer: Knowledge Injection in Transformer Feed-Forward Layers. (arXiv:2201.05742v2 [cs.CL] UPDATED)

Recent days have witnessed a diverse set of knowledge injection models for
pre-trained language models (PTMs); however, most previous studies neglect the
PTMs' own ability with quantities of implicit knowledge stored in parameters. A
recent study has observed knowledge neurons in the Feed Forward Network (FFN),
which are responsible for expressing factual knowledge. In this work, we
propose a simple model, Kformer, which takes advantage of the knowledge stored
in PTMs and external knowledge via knowledge injection in Transformer FFN …

arxiv knowledge transformer