Training Language Models with Memory Augmentation. (arXiv:2205.12674v1 [cs.CL])

Recent work has improved language models remarkably by equipping them with a
non-parametric memory component. However, most existing approaches only
introduce memories at testing time, or represent them using a separately
trained encoder -- resulting in sub-optimal training of the language model. In
this work, we present TRIME, a novel yet simple training approach designed for
training language models with memory augmentation. Our approach uses a training
objective that directly takes in-batch examples as accessible memory. We also
present new …

arxiv augmentation language language models memory training