GRASP: A novel benchmark for evaluating language GRounding And Situated Physics understanding in multimodal language models

This paper presents GRASP, a novel benchmark to evaluate the language grounding and physical understanding capabilities of video-based multimodal large language models (LLMs). This evaluation is accomplished via a two-tier approach leveraging Unity simulations. The first level tests for language grounding by assessing a model's ability to relate simple textual descriptions with visual information. The second level evaluates the model's understanding of "Intuitive Physics" principles, such as object permanence and continuity. In addition to releasing the benchmark, we use it …

benchmark capabilities cs.cl evaluation language language models large language large language models llms multimodal novel paper physics simulations tests understanding unity via video