Visually Guided Generative Text-Layout Pre-training for Document Intelligence

arXiv:2403.16516v1 Announce Type: cross
Abstract: Prior study shows that pre-training techniques can boost the performance of visual document understanding (VDU), which typically requires models to gain abilities to perceive and reason both document texts and layouts (e.g., locations of texts and table-cells). To this end, we propose visually guided generative text-layout pre-training, named ViTLP. Given a document image, the model optimizes hierarchical language and layout modeling objectives to generate the interleaved text and layout sequence. In addition, to address the …

abstract arxiv boost cells cs.cl cs.cv document document understanding generative intelligence locations performance pre-training prior reason shows study table text training type understanding visual