Feb. 28, 2024, 10:59 a.m. | /u/SunsetOneSix

Machine Learning www.reddit.com

**Paper**: [https://arxiv.org/abs/2402.08562](https://arxiv.org/abs/2402.08562)

**Code**: [https://github.com/GCYZSL/MoLA](https://github.com/GCYZSL/MoLA)

**Abstract**:

>Parameter-efficient tuning (PEFT) techniques like low-rank adaptation (LoRA) offer training efficiency on Large Language Models, but their impact on model performance remains limited. Recent efforts integrate LoRA and Mixture-of-Experts (MoE) to improve the performance of PEFT methods. Despite promising results, research on improving the efficiency of LoRA with MoE is still in its early stages. Recent studies have shown that experts in the MoE architecture have different strengths and also exhibit some redundancy. Does this …

abstract machinelearning

Data Scientist (m/f/x/d)

@ Symanto Research GmbH & Co. KG | Spain, Germany

Autonomy Intern - Motion Planning

@ Fox Robotics | Austin, Texas

Data Analyst Intern m/f/d - Content

@ Deezer | Paris, France

Senior Machine Learning Engineer

@ Logic20/20 Inc. | Oakland, CA, United States

Data Analyst (Tableau/BI/CRMA)

@ Databricks | Bengaluru, India

Data Engineer

@ Octopus | London, United Kingdom