[D] PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns

Dataset: [https://github.com/declare-lab/LLM-PuzzleTest/tree/master/PuzzleVQA](https://github.com/declare-lab/LLM-PuzzleTest/tree/master/PuzzleVQA)

Paper: [https://arxiv.org/abs/2403.13315](https://arxiv.org/abs/2403.13315)

Large multimodal models extend the impressive capabilities of large language models by integrating multimodal understanding abilities. However, it is not clear how they can emulate the general intelligence and reasoning ability of humans. As recognizing patterns and abstracting concepts are key to general intelligence, we introduce PuzzleVQA, a collection of puzzles based on abstract patterns. With this dataset, we evaluate large multimodal models with abstract patterns based on fundamental concepts, including colors, numbers, sizes, and shapes. …

abstract capabilities clear collection concepts dataset general however humans intelligence key language language models large language large language models large multimodal models machinelearning multimodal multimodal models patterns reasoning understanding