Investigating the Effectiveness of HyperTuning via Gisting

arXiv:2402.16817v1 Announce Type: new
Abstract: Gisting (Mu et al., 2023) is a simple method for training models to compress information into fewer token representations using a modified attention mask, and can serve as an economical approach to training Transformer-based hypernetworks. We introduce HyperLlama, a set of Gisting-based hypernetworks built on Llama-2 models that generates task-specific soft prefixes based on few-shot inputs. In experiments across P3, Super-NaturalInstructions and Symbol Tuning datasets, we show that HyperLlama models can effectively compress information from …

abstract arxiv attention cs.cl information llama serve set simple token training training models transformer type via