LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding

arXiv:2404.05225v1 Announce Type: new
Abstract: Recently, leveraging large language models (LLMs) or multimodal large language models (MLLMs) for document understanding has been proven very promising. However, previous works that employ LLMs/MLLMs for document understanding have not fully explored and utilized the document layout information, which is vital for precise document understanding. In this paper, we propose LayoutLLM, an LLM/MLLM based method for document understanding. The core of LayoutLLM is a layout instruction tuning strategy, which is specially designed to enhance …

arxiv cs.cl cs.cv document document understanding language language models large language large language models type understanding