Evaluating Text Summaries Generated by Large Language Models Using OpenAI's GPT

arXiv:2405.04053v1 Announce Type: cross
Abstract: This research examines the effectiveness of OpenAI's GPT models as independent evaluators of text summaries generated by six transformer-based models from Hugging Face: DistilBART, BERT, ProphetNet, T5, BART, and PEGASUS. We evaluated these summaries based on essential properties of high-quality summary - conciseness, relevance, coherence, and readability - using traditional metrics such as ROUGE and Latent Semantic Analysis (LSA). Uniquely, we also employed GPT not as a summarizer but as an evaluator, allowing it to …

abstract arxiv bart bert cs.ai cs.cl cs.lg face generated gpt gpt models hugging face independent language language models large language large language models openai quality research six summary text transformer transformer-based models type