Efficient End-to-End Visual Document Understanding with Rationale Distillation

arXiv:2311.09612v2 Announce Type: replace-cross
Abstract: Understanding visually situated language requires interpreting complex layouts of textual and visual elements. Pre-processing tools, such as optical character recognition (OCR), can map document image inputs to textual tokens, then large language models (LLMs) can reason over text. However, such methods have high computational and engineering complexity. Can small pretrained image-to-text models accurately understand visual documents through similar recognition and reasoning steps instead? We propose Rationale Distillation (RD), which incorporates the outputs of OCR tools, …

abstract arxiv character recognition complexity computational cs.cl cs.cv distillation document document understanding engineering however image inputs language language models large language large language models llms map ocr optical optical character recognition pre-processing processing reason recognition text textual tokens tools type understanding visual