SIBO: A Simple Booster for Parameter-Efficient Fine-Tuning

arXiv:2402.11896v1 Announce Type: new
Abstract: Fine-tuning all parameters of large language models (LLMs) necessitates substantial computational power and extended time. Latest advancements in parameter-efficient fine-tuning (PEFT) techniques, such as Adapter tuning and LoRA, allow for adjustments to only a minor fraction of the parameters of these LLMs. Concurrently, it has been noted that the issue of over-smoothing diminishes the effectiveness of these Transformer-based LLMs, resulting in suboptimal performances in downstream tasks. In this paper, we present SIBO, which is a …

abstract arxiv computational cs.cl fine-tuning language language models large language large language models llms lora parameters peft power simple type