SLPL SHROOM at SemEval\-2024 Task 06: A comprehensive study on models ability to detect hallucination

arXiv:2404.04845v1 Announce Type: new
Abstract: Language models, particularly generative models, are susceptible to hallucinations, generating outputs that contradict factual knowledge or the source text. This study explores methods for detecting hallucinations in three SemEval-2024 Task 6 tasks: Machine Translation, Definition Modeling, and Paraphrase Generation. We evaluate two methods: semantic similarity between the generated text and factual references, and an ensemble of language models that judge each other's outputs. Our results show that semantic similarity achieves moderate accuracy and correlation scores …

abstract arxiv cs.ai cs.cl definition generative generative models hallucination hallucinations knowledge language language models machine machine translation modeling study tasks text translation type