Dual Modalities of Text: Visual and Textual Generative Pre-training

arXiv:2404.10710v1 Announce Type: new
Abstract: Harnessing visual texts represents a burgeoning frontier in the evolution of language modeling. In this paper, we introduce a novel pre-training framework for a suite of pixel-based autoregressive language models, pre-training on a corpus of over 400 million documents rendered as RGB images. Our approach is characterized by a dual-modality training regimen, engaging both visual data through next patch prediction with a regression head and textual data via next token prediction with a classification head. …

abstract arxiv autoregressive cs.cl cs.cv documents evolution framework generative images language language models modeling novel paper pixel pre-training text textual training type visual