DesignQA: A Multimodal Benchmark for Evaluating Large Language Models' Understanding of Engineering Documentation

arXiv:2404.07917v1 Announce Type: cross
Abstract: This research introduces DesignQA, a novel benchmark aimed at evaluating the proficiency of multimodal large language models (MLLMs) in comprehending and applying engineering requirements in technical documentation. Developed with a focus on real-world engineering challenges, DesignQA uniquely combines multimodal data-including textual design requirements, CAD images, and engineering drawings-derived from the Formula SAE student competition. Different from many existing MLLM benchmarks, DesignQA contains document-grounded visual questions where the input image and input document come from different …

arxiv benchmark cs.ai cs.cl documentation engineering language language models large language large language models multimodal type understanding