Language-Independent Representations Improve Zero-Shot Summarization

arXiv:2404.05720v1 Announce Type: new
Abstract: Finetuning pretrained models on downstream generation tasks often leads to catastrophic forgetting in zero-shot conditions. In this work, we focus on summarization and tackle the problem through the lens of language-independent representations. After training on monolingual summarization, we perform zero-shot transfer to new languages or language pairs. We first show naively finetuned models are highly language-specific in both output behavior and internal representations, resulting in poor zero-shot performance. Next, we propose query-key (QK) finetuning to …

abstract arxiv catastrophic forgetting cs.ai cs.cl finetuning focus independent language languages leads lens pretrained models show summarization tasks through training transfer type work zero-shot