Evaluating Dialect Robustness of Language Models via Conversation Understanding

arXiv:2405.05688v1 Announce Type: new
Abstract: With an evergrowing number of LLMs reporting superlative performance for English, their ability to perform equitably for different dialects of English (i.e., dialect robustness) needs to be ascertained. Specifically, we use English language (US English or Indian English) conversations between humans who play the word-guessing game of `taboo'. We formulate two evaluative tasks: target word prediction (TWP) (i.e.predict the masked target word in a conversation) and target word selection (TWS) (i.e., select the most likely …

abstract arxiv conversation conversations cs.cl english english language game humans indian language language models llms performance reporting robustness type understanding via word