Musketeer: Joint Training for Multi-task Vision Language Model with Task Explanation Prompts

arXiv:2305.07019v2 Announce Type: replace
Abstract: We present a vision-language model whose parameters are jointly trained on all tasks and fully shared among multiple heterogeneous tasks which may interfere with each other, resulting in a single model which we named Musketeer. The integration of knowledge across heterogeneous tasks is enabled by a novel feature called Task Explanation Prompt (TEP). With rich and structured information such as task input/output format, TEP reduces interference among tasks, allowing the model to focus on their …

abstract arxiv cs.ai cs.cl cs.cv integration knowledge language language model multiple parameters prompts tasks training type vision vision language model