Improving Small Language Models' Mathematical Reasoning via Equation-of-Thought Distillation

This work addresses the challenge of democratizing advanced Large Language Models (LLMs) by compressing their mathematical reasoning capabilities into sub-billion parameter Small Language Models (SLMs) without compromising performance. We introduce Equation-of-Thought Distillation (EoTD), a novel technique that encapsulates the reasoning process into equation-based representations to construct an EoTD dataset for fine-tuning SLMs. Additionally, we propose the Ensemble Thoughts Distillation (ETD) framework to enhance the reasoning performance of SLMs. This involves creating a reasoning dataset with multiple thought processes, including Chain-of-Thought …

advanced billion capabilities challenge construct cs.ai cs.cl dataset distillation equation fine-tuning language language models large language large language models llms mathematical reasoning novel performance process reasoning slms small small language models thought via work