Tables as Images? Exploring the Strengths and Limitations of LLMs on Multimodal Representations of Tabular Data

arXiv:2402.12424v1 Announce Type: new
Abstract: In this paper, we investigate the effectiveness of various LLMs in interpreting tabular data through different prompting strategies and data formats. Our analysis extends across six benchmarks for table-related tasks such as question-answering and fact-checking. We introduce for the first time the assessment of LLMs' performance on image-based table representations. Specifically, we compare five text-based and three image-based table representations, demonstrating the influence of representation and prompting on LLM performance. Our study provides insights into …

abstract analysis arxiv benchmarks cs.ai cs.cl cs.cv cs.lg data fact-checking images limitations llms multimodal paper prompting question six strategies table tables tabular tabular data tasks through type