Low-Rank Adaptation for Multilingual Summarization: An Empirical Study

arXiv:2311.08572v2 Announce Type: replace-cross
Abstract: Although the advancements of pre-trained Large Language Models have significantly accelerated recent progress in NLP, their ever-increasing size poses significant challenges for conventional fine-tuning, especially in memory-intensive tasks. We investigate the potential of Parameter-Efficient Fine-Tuning, focusing on Low-Rank Adaptation (LoRA), in the domain of multilingual summarization, a task that is both challenging (due to typically long inputs), and relatively unexplored. We conduct an extensive study across different data availability scenarios, including high- and low-data settings, …

abstract arxiv challenges cs.ai cs.cl cs.lg domain fine-tuning language language models large language large language models lora low low-rank adaptation memory multilingual nlp progress study summarization tasks type