CM3: A Causal Masked Multimodal Model of the Internet. (arXiv:2201.07520v1 [cs.CL])

We introduce CM3, a family of causally masked generative models trained over
a large corpus of structured multi-modal documents that can contain both text
and image tokens. Our new causally masked approach generates tokens left to
right while also masking out a small number of long token spans that are
generated at the end of the string, instead of their original positions. The
casual masking object provides a type of hybrid of the more common causal and
masked language models, …

arxiv internet multimodal