Teaching a Multilingual Large Language Model to Understand Multilingual Speech via Multi-Instructional Training

arXiv:2404.10922v1 Announce Type: new
Abstract: Recent advancements in language modeling have led to the emergence of Large Language Models (LLMs) capable of various natural language processing tasks. Despite their success in text-based tasks, applying LLMs to the speech domain remains limited and challenging. This paper presents BLOOMZMMS, a novel model that integrates a multilingual LLM with a multilingual speech encoder, aiming to harness the capabilities of LLMs for speech recognition and beyond. Utilizing a multi-instructional training approach, we demonstrate the …

abstract arxiv cs.cl cs.sd domain eess.as emergence language language model language models language processing large language large language model large language models llms modeling multilingual natural natural language natural language processing paper processing speech success tasks teaching text training type via