This AI Paper from Snowflake evaluated the performance of GPT-4 on Document Understanding tasks, and it seems even models of this parameter count underperform in image-only setup and vastly benefit from providing text in addition to the input image.

Researchers from Snowflake evaluated various configurations of GPT-4 models, including integrating external OCR engines with document images. This approach aims to enhance document understanding by combining OCR-recognized text with visual inputs, allowing the models to simultaneously process both types of information. The study examined different versions of GPT-4, such as the TURBO V model, which supports high-resolution images and extensive context windows up to 128k tokens, enabling it to handle complex documents more effectively.

The proposed method was evaluated using …

ai paper benefit count document document understanding gpt gpt-4 image images input machinelearningnews ocr paper performance researchers setup snowflake tasks text understanding