FedEval-LLM: Federated Evaluation of Large Language Models on Downstream Tasks with Collective Wisdom

arXiv:2404.12273v1 Announce Type: cross
Abstract: Federated Learning (FL) has emerged as a promising solution for collaborative training of large language models (LLMs). However, the integration of LLMs into FL introduces new challenges, particularly concerning the evaluation of LLMs. Traditional evaluation methods that rely on labeled test sets and similarity-based metrics cover only a subset of the acceptable answers, thereby failing to accurately reflect the performance of LLMs on generative tasks. Meanwhile, although automatic evaluation methods that leverage advanced LLMs present …

abstract arxiv challenges collaborative collective cs.ai cs.cl cs.lg evaluation federated learning however integration language language models large language large language models llm llms solution tasks test training type