Deal, or no deal (or who knows)? Forecasting Uncertainty in Conversations using Large Language Models

Effective interlocutors account for the uncertain goals, beliefs, and emotions of others. But even the best human conversationalist cannot perfectly anticipate the trajectory of a dialogue. How well can language models represent inherent uncertainty in conversations? We propose FortUne Dial, an expansion of the long-standing "conversation forecasting" task: instead of just accuracy, evaluation is conducted with uncertainty-aware metrics, effectively enabling abstention on individual instances. We study two ways in which language models potentially represent outcome uncertainty (internally, using scores and …

conversations cs.ai cs.cl cs.lg deal dialogue emotions expansion forecasting human language language models large language large language models trajectory uncertain uncertainty