Evaluating Large Language Models as Generative User Simulators for Conversational Recommendation

arXiv:2403.09738v1 Announce Type: new
Abstract: Synthetic users are cost-effective proxies for real users in the evaluation of conversational recommender systems. Large language models show promise in simulating human-like behavior, raising the question of their ability to represent a diverse population of users. We introduce a new protocol to measure the degree to which language models can accurately emulate human behavior in conversational recommendation. This protocol is comprised of five tasks, each designed to evaluate a key property that a synthetic …

abstract arxiv behavior conversational cost cs.ai cs.cl cs.ir diverse evaluation generative human human-like language language models large language large language models population protocol proxies question recommendation recommender systems show synthetic systems type