Can LLMs Converse Formally? Automatically Assessing LLMs in Translating and Interpreting Formal Specifications

arXiv:2403.18327v1 Announce Type: new
Abstract: Stakeholders often describe system requirements using natural language which are then converted to formal syntax by a domain-expert leading to increased design costs. This paper assesses the capabilities of Large Language Models (LLMs) in converting between natural language descriptions and formal specifications. Existing work has evaluated the capabilities of LLMs in generating formal syntax such as source code but such experiments are typically hand-crafted and use problems that are likely to be in the training …

abstract arxiv capabilities costs cs.ai cs.cl design domain expert language language models large language large language models llms natural natural language paper requirements stakeholders syntax type