Can Your Model Tell a Negation from an Implicature? Unravelling Challenges With Intent Encoders

arXiv:2403.04314v1 Announce Type: new
Abstract: Conversational systems often rely on embedding models for intent classification and intent clustering tasks. The advent of Large Language Models (LLMs), which enable instructional embeddings allowing one to adjust semantics over the embedding space using prompts, are being viewed as a panacea for these downstream conversational tasks. However, traditional evaluation benchmarks rely solely on task metrics that don't particularly measure gaps related to semantic understanding. Thus, we propose an intent semantic toolkit that gives a …

abstract arxiv challenges classification clustering conversational cs.cl embedding embedding models embeddings language language models large language large language models llms prompts semantics space systems tasks type