Contextual Categorization Enhancement through LLMs Latent-Space

arXiv:2404.16442v1 Announce Type: new
Abstract: Managing the semantic quality of the categorization in large textual datasets, such as Wikipedia, presents significant challenges in terms of complexity and cost. In this paper, we propose leveraging transformer models to distill semantic information from texts in the Wikipedia dataset and its associated categories into a latent space. We then explore different approaches based on these encodings to assess and enhance the semantic identity of the categories. Our graphical approach is powered by Convex …

abstract arxiv challenges complexity cost cs.ai cs.cl dataset datasets information latent-space llms paper quality semantic space terms textual through transformer transformer models type wikipedia