MEND: Meta dEmonstratioN Distillation for Efficient and Effective In-Context Learning

arXiv:2403.06914v1 Announce Type: new
Abstract: Large Language models (LLMs) have demonstrated impressive in-context learning (ICL) capabilities, where a LLM makes predictions for a given test input together with a few input-output pairs (demonstrations). Nevertheless, the inclusion of demonstrations leads to a quadratic increase in the computational overhead of the self-attention mechanism. Existing solutions attempt to distill lengthy demonstrations into compact vectors. However, they often require task-specific retraining or compromise LLM's in-context learning performance. To mitigate these challenges, we present Meta …

abstract arxiv attention capabilities computational context cs.ai cs.cl distillation inclusion in-context learning input-output language language models large language large language models leads llm llms meta predictions self-attention test together type