AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs

arXiv:2311.06753v2 Announce Type: replace
Abstract: In this work, we extend the instruction-tuned Llama-2 model with end-to-end general-purpose speech processing and reasoning abilities while maintaining the wide range of original LLM capabilities, without using any carefully curated paired data. The resulting end-to-end model, named AudioChatLlama, can utilize audio prompts as a replacement for text and sustain a conversation. Such a model also has extended cross-modal capabilities such as being able to perform spoken question answering (QA), speech translation, and audio summarization …

abstract arxiv audio capabilities cs.ai cs.cl data general instruction-tuned llama llm llms processing prompts reasoning replacement speech speech processing text type work