Large Language Models "Ad Referendum": How Good Are They at Machine Translation in the Legal Domain?

This study evaluates the machine translation (MT) quality of two state-of-the-art large language models (LLMs) against a tradition-al neural machine translation (NMT) system across four language pairs in the legal domain. It combines automatic evaluation met-rics (AEMs) and human evaluation (HE) by professional transla-tors to assess translation ranking, fluency and adequacy. The re-sults indicate that while Google Translate generally outperforms LLMs in AEMs, human evaluators rate LLMs, especially GPT-4, comparably or slightly better in terms of producing contextually adequate and …

art cs.ai cs.cl domain evaluation good human language language models large language large language models legal llms machine machine translation neural machine translation professional quality referendum state study tradition translation