Differentially Private Training of Mixture of Experts Models

This position paper investigates the integration of Differential Privacy (DP) in the training of Mixture of Experts (MoE) models within the field of natural language processing. As Large Language Models (LLMs) scale to billions of parameters, leveraging expansive datasets, they exhibit enhanced linguistic capabilities and emergent abilities. However, this growth raises significant computational and privacy concerns. Our study addresses these issues by exploring the potential of MoE models, known for their computational efficiency, and the application of DP, a standard …

capabilities cs.cr cs.lg datasets differential differential privacy experts growth integration language language models language processing large language large language models llms mixture of experts moe natural natural language natural language processing paper parameters privacy processing raises scale training