Exploring the Limits of Fine-grained LLM-based Physics Inference via Premise Removal Interventions

arXiv:2404.18384v1 Announce Type: new
Abstract: Language models can hallucinate when performing complex and detailed mathematical reasoning. Physics provides a rich domain for assessing mathematical reasoning capabilities where physical context imbues the use of symbols which needs to satisfy complex semantics (\textit{e.g.,} units, tensorial order), leading to instances where inference may be algebraically coherent, yet unphysical. In this work, we assess the ability of Language Models (LMs) to perform fine-grained mathematical and physical reasoning using a curated dataset encompassing multiple notations …

abstract arxiv capabilities context cs.cl domain fine-grained inference instances language language models llm mathematical reasoning physics reasoning semantics type units via