One Model, Multiple Modalities: A Sparsely Activated Approach for Text, Sound, Image, Video and Code. (arXiv:2205.06126v1 [cs.CL])

Web: http://arxiv.org/abs/2205.06126

People perceive the world with multiple senses (e.g., through hearing sounds,
reading words and seeing objects). However, most existing AI systems only
process an individual modality. This paper presents an approach that excels at
handling multiple modalities of information with a single model. In our
"{SkillNet}" model, different parts of the parameters are specialized for
processing different modalities. Unlike traditional dense models that always
activate all the model parameters, our model sparsely activates parts of the
parameters whose skills are …

arxiv code image model sound text video