Mastering Text, Code and Math Simultaneously via Fusing Highly Specialized Language Models

arXiv:2403.08281v1 Announce Type: new
Abstract: Underlying data distributions of natural language, programming code, and mathematical symbols vary vastly, presenting a complex challenge for large language models (LLMs) that strive to achieve high performance across all three domains simultaneously. Achieving a very high level of proficiency for an LLM within a specific domain often requires extensive training with relevant corpora, which is typically accompanied by a sacrifice in performance in other domains. In this paper, we propose to fuse models that …

abstract arxiv challenge code cs.ai cs.cl data domains language language models large language large language models llm llms math natural natural language performance presenting programming text type via