Reasoning before Comparison: LLM-Enhanced Semantic Similarity Metrics for Domain Specialized Text Analysis

arXiv:2402.11398v1 Announce Type: new
Abstract: In this study, we leverage LLM to enhance the semantic analysis and develop similarity metrics for texts, addressing the limitations of traditional unsupervised NLP metrics like ROUGE and BLEU. We develop a framework where LLMs such as GPT-4 are employed for zero-shot text identification and label generation for radiology reports, where the labels are then used as measurements for text similarity. By testing the proposed framework on the MIMIC data, we find that GPT-4 generated …

abstract analysis arxiv bleu comparison cs.ai cs.cl domain framework gpt gpt-4 limitations llm llms metrics nlp reasoning semantic study text type unsupervised zero-shot