MemoryPrompt: A Light Wrapper to Improve Context Tracking in Pre-trained Language Models

arXiv:2402.15268v1 Announce Type: cross
Abstract: Transformer-based language models (LMs) track contextual information through large, hard-coded input windows. We introduce MemoryPrompt, a leaner approach in which the LM is complemented by a small auxiliary recurrent network that passes information to the LM by prefixing its regular input with a sequence of vectors, akin to soft prompts, without requiring LM finetuning. Tested on a task designed to probe a LM's ability to keep track of multiple fact updates, a MemoryPrompt-augmented LM outperforms …

abstract arxiv context cs.ai cs.cl cs.lg information language language models light lms network small through tracking transformer type windows wrapper