BLT: Can Large Language Models Handle Basic Legal Text?

arXiv:2311.09693v2 Announce Type: replace
Abstract: We find that the best publicly available LLMs like GPT-4, Claude, and {PaLM 2} currently perform poorly at basic legal text handling. We introduce a benchmark consisting of tasks that lawyers and paralegals would expect LLMs to handle zero-shot, such as looking up the text at a line of a witness deposition or at a subsection of a contract. LLMs' poor performance on this benchmark casts into doubt their reliability as-is for legal practice. However, …

abstract arxiv basic benchmark claude cs.ai cs.cl expect gpt gpt-4 language language models large language large language models lawyers legal llms palm palm 2 tasks text type zero-shot