MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response

arXiv:2309.08730v3 Announce Type: replace-cross
Abstract: Large Language Models (LLMs) have shown immense potential in multimodal applications, yet the convergence of textual and musical domains remains not well-explored. To address this gap, we present MusiLingo, a novel system for music caption generation and music-related query responses. MusiLingo employs a single projection layer to align music representations from the pre-trained frozen music audio model MERT with a frozen LLM, bridging the gap between music audio and textual contexts. We train it on …

abstract applications arxiv captioning convergence cs.ai cs.cl cs.mm cs.sd domains eess.as gap language language models large language large language models llms multimodal music novel query responses text textual type