Emu: Generative Pretraining in Multimodality

arXiv:2307.05222v2 Announce Type: replace
Abstract: We present Emu, a Transformer-based multimodal foundation model, which can seamlessly generate images and texts in multimodal context. This omnivore model can take in any single-modality or multimodal data input indiscriminately (e.g., interleaved image, text and video) through a one-model-for-all autoregressive training process. First, visual signals are encoded into embeddings, and together with text tokens form an interleaved input sequence. Emu is then end-to-end trained with a unified objective of classifying the next text token …

abstract arxiv autoregressive context cs.cv data embeddings emu foundation foundation model generate generative image images multimodal multimodal data multimodality pretraining process text through training transformer type video visual